Add capability to parse nested suites in capgen #691

climbfuji · 2025-10-07T18:42:38Z

Description

This PR adds the capability to parse nested suites in capgen, as discussed in #275. The implementation supports two types of nested suites using the XML element nested_suite.

> cat test/nested_suite_test/main_suite.xml
<?xml version="1.0" encoding="UTF-8"?>

<suites version="2.0">

  <suite name="radiation3_suite">
    <group name="rad_lw_group">
      <scheme>rad_lw</scheme>
    </group>
    <group name="rad_sw_group">
      <nested_suite name="radiation3_subsuite" group="rad_sw_group" file="radiation3_subsuite.xml"/>
    </group>
  </suite>

  <suite name="main_suite">
    <group name="radiation1">
      <subcycle loop="num_subcycles_for_effr">
        <scheme>effr_pre</scheme>
        <subcycle loop="2">
          <subcycle loop="2">
            <scheme>effr_calc</scheme>
          </subcycle>
        </subcycle>
        <scheme>effr_post</scheme>
      </subcycle>
      <nested_suite name="radiation2_suite" group="effrs_calc" file="radiation2_suite.xml"/>
    </group>
    <nested_suite name="radiation3_suite"/>
  </suite>

</suites>

Notes/Features

The nested suites can reside in the same file or external xml files, and the suites can be nested recursively.
All external xml files must be valid under the new XML schema 2.0 that was added to support multiple suites within a suite definition file under a root <suites> element.
Old suite definition files using XML schema 1.0 are still supported, but nested suites are only support for schema version 2 and above.
Nested suites inside a group construct must have the group= attribute; the content of that group in the referenced suite is merged into the existing group while preserving the order. This is the first example (aka the merge=true example) described in Add suite keyword to suite definition file? #275 (comment).
Nested suites one level up, i.e. at the group level, must not have the group= attribute; the content of the entire referenced suite (i.e. all groups) is added in the correct location in the main suite. This is the second example (aka the merge=false example) described in Add suite keyword to suite definition file? #275 (comment).
There is no limit on how many nested suites can be referenced in a suite at either of the two levels described in 4 and 5.
Nested suites can be recursive, and external xml files containing nested suites can reference additional suites in the same file or in external files.
Disclaimer I don't like writing docstrings and doctests, therefore I asked chatgpt to do that for me. It did a decent job, certainly better than I would have, but I still had to go in and correct a few things.
The PrettyElementTree class was replaced by a much shorter function write_xml_file that uses features available in the already-used XML python library. And since it contained dom twice in the name, I simply couldn't resist! The output of the new implementation is almost identical to the previous PrettyElementTree output, see screenshot at the bottom of the PR description.
More than 2/3 of the changed/added files in this PR are just for the new test nested_suite_test, thus the actual changes to the code are quite small and hopefully easy to review.

User interface changes?: Yes (sort of), but these are optional. In order to use the new functionality, users have to update their suite definition file to the new XML schema version 2.0 (including all XML files that contain nested suites) and use the above syntax for nested_suite elements. There are no user interface changes for the previous XML schema 1.0, which remains valid, and there are no user interface changes for invoking capgen or in the auto-generated code.

Issues

Fixes #275

Testing

Test removed: none
Test added:
- added test/nested_suite_test - see README.md in that directory for more information
- added doctests for the new functions in xml_tools.py
Unit tests: all pass
System tests: all pass
Manual testing: nested_suite_test

Additional information

Difference in datatable.xml output of the new write_xml_file function (left) and the old PrettyElementTree class (right):

In a nutshell, the differences are no white spaces before the closing characters of XML elements, and the additional <? xml ...?> line at the top. Both of these seem reasonable to me.

…schema

…pp_suite.py, scripts/parse_tools/xml_tools.py, and test/nested_suite_test/test_host.F90

… feature/nested_suites

mkavulich

I have a few questions

scripts/ccpp_suite.py

mkavulich · 2025-10-22T17:00:37Z

scripts/parse_tools/xml_tools.py

+        ... </suites>
+        ... '''
+        >>> root = ET.fromstring(xml)
+        >>> expand_nested_suites(root, logger)


I'm curious if you've tested the behavior of infinite recursion, is that something we should have a test for to make sure it fails gracefully?

No, and I recall that we discussed this two weeks ago. If I remember correctly, we assumed that Python would throw an error about the recursion (but we didn't try). We could make a doctest for this, though.

I wonder as part of this algorithm if it would make sense to keep a list of nested suites that have been processed that contain another nested suite. Couldn't you avoid infinite recursion by stopping if you're currently trying to processes a nested suite that already occurs in the list of already-processed nested suites that contain nested suites?

Implemented in 0d144ae

I'm not happy with the current solution, it gives no hints as to where the problem might lie.

How about we use the current solution (since it is the safest and easiest one), but collect the names of the suites and just spit out that list when we throw the exception?

test/nested_suite_test/ccpp_kinds.F90

test/nested_suite_test/main_suite.xml

test/nested_suite_test/test_host.F90

grantfirl

This is slick. I tried to review in order to understand how to use the new functionality and the implementation in the python scripts. I didn't really review the added test.

peverwhee

a couple of requests. I haven't done a full test review yet but will get to that in my second review!

scripts/ccpp_suite.py

scripts/parse_tools/xml_tools.py

… feature/nested_suites

… absolute or relative, always write expanded SDF

…ccpp-framework into feature/nested_suites

gold2718 · 2025-11-10T11:40:42Z

Nested suites one level up, i.e. at the group level, must not have the group= attribute

Why is this requirement needed or desired?

gold2718 · 2025-11-10T18:14:30Z

Is it okay for a SDF to have more than one suite after expansion of the nested suites?
If this is true, why are suites which happen to be expanded into a different suite in the same SD removed? Would it be illegal somehow to keep both?
It is certainly less fragile to remove them (as opposed to generating a suite cap for a subsuite), however, the current behavior will need to be clearly documented.

climbfuji · 2025-11-10T22:02:31Z

Is it okay for a SDF to have more than one suite after expansion of the nested suites? If this is true, why are suites which happen to be expanded into a different suite in the same SD removed? Would it be illegal somehow to keep both? It is certainly less fragile to remove them (as opposed to generating a suite cap for a subsuite), however, the current behavior will need to be clearly documented.

Since you are still awake ;-)

The removal of the expanded suite happens entirely in memory and ONLY within the expansion of the top-level suite you are processing. That means if you have two suites A and B that capgen is supposed to process, and both refer to the same nested suite, they will both see the original, unmodified suites as they are on the file system.

The removal was at least for the reason of writing the fully expanded suite to disk and to the datatable (so that it is clear that this is the final suite that is getting processed). I don't recall immediately if there was another reason for it - it's been too long ago that I worked on the code.

gold2718 · 2025-11-11T11:36:46Z

The removal of the expanded suite happens entirely in memory and ONLY within the expansion of the top-level suite you are processing.

I do not have any issue with this feature, removing a suite which is nested inside another suite in the same file seems correct.

However, that is not my question. As an example, say you have an SDF which has two suites with neither being nested. I think we end up with two suites. Is that correct? Does / should capgen handle that situation as if these suites were in two different SDFs?

climbfuji · 2025-11-12T16:15:57Z

@gold2718 Ignore all of what I wrote at an apparently too early time today. PEBKAC.

I do see this when I process multiple suites in one SDF that remain after expansion:

      if (trim(suite_name) == 'main_suite') then
         call main_suite_timestep_initial(errflg=errflg, errmsg=errmsg)
      else if (trim(suite_name) == 'another_suite') then
         call another_suite_timestep_initial(errflg=errflg, errmsg=errmsg)
      else
         write(errmsg, '(3a)')"No suite named ", trim(suite_name), "found"
         errflg = 1
      end if

But questions remain. Do we want this? Should there be only one remaining suite after expanding suites?

The more I think about it, I get the impression that it would be easier to go back to just one suite per SDF.

…he suite level

climbfuji · 2025-11-12T17:37:32Z

@gold2718 I implemented the capability to insert a specific group at the suite level, as per #691 (comment) and the discussion we had at the tag up on Monday.

Commit:

commit f336c174b618817bdcaee84abe3db65dd43d96ea (HEAD -> feature/nested_suites)
Author: Dom Heinzeller <[email protected]>
Date:   Wed Nov 12 10:36:19 2025 -0700

    Add capability to insert only a specific group of a nested suite at the suite level

The output for the nested_suite_test is:

> cat test/nested_suite_test/ccpp/main_suite_expanded.xml
<?xml version="1.0" ?>
<suites version="2.0">
  <suite name="main_suite">
    <group name="radiation1">
      <subcycle loop="num_subcycles_for_effr">
        <scheme>effr_pre</scheme>
        <subcycle loop="2">
          <subcycle loop="2">
            <scheme>effr_calc</scheme>
          </subcycle>
        </subcycle>
        <scheme>effr_post</scheme>
      </subcycle>
      <subcycle loop="num_subcycles_for_effr">
        <scheme>effrs_calc</scheme>
      </subcycle>
      <scheme>effr_diag</scheme>
    </group>
    <group name="rad_lw_group">
      <scheme>rad_lw</scheme>
    </group>
    <group name="rad_sw_group">
      <scheme>rad_sw</scheme>
    </group>
  </suite>
</suites>

gold2718 · 2025-11-12T19:23:41Z

The more I think about it, I get the impression that it would be easier to go back to just one suite per SDF.

I would support this. We could drop the suites tag, add the version back to suite, and just add the nested_suite tag. Thoughts @peverwhee, @grantfirl, @mkavulich, @dustinswales (fingers crossed on that last mention)?

peverwhee · 2025-11-13T17:15:58Z

The more I think about it, I get the impression that it would be easier to go back to just one suite per SDF.

I would support this. We could drop the suites tag, add the version back to suite, and just add the nested_suite tag. Thoughts @peverwhee, @grantfirl, @mkavulich, @dustinswales (fingers crossed on that last mention)?

I agree - one suite per SDF sounds simplest.

…ion of nested suites

climbfuji · 2025-11-13T19:37:39Z

@peverwhee @gold2718 I reverted back to only one suite per SDF. The code is so much cleaner! The only things that got a little messy are the doctests, because I need to read nested suites from files (I could have tried Python mock etc, but this seemed easy enough). I also implemented a - what I believe - robust way of catching recursions.

mkavulich · 2025-11-14T18:49:50Z

@climbfuji So if I'm understanding right, the difference between this latest implementation and the original is that we allow nested suites, but only if they are defined in another file?

climbfuji · 2025-11-14T19:29:12Z

@climbfuji So if I'm understanding right, the difference between this latest implementation and the original is that we allow nested suites, but only if they are defined in another file?

Correct. That eliminates the need for <suites> and make schema v2 much simpler.

mkavulich

Looks good to me (again)

peverwhee

thanks @climbfuji !

SIMA will eventually need an offline way to generate the expanded SDF because of the way the build system works, but I'll open an issue and make a PR for that in the future!

gold2718

I have not done a complete code review but I wrote some tests instead. I will submit my changes and new test files as a PR to this branch.
I think all the tests should pass, we should discuss any with which you disagree.

gold2718 · 2025-10-09T20:43:06Z

schema/suite_v2_0.xsd

+        </xs:choice>
+      </xs:sequence>
+      <xs:attribute name="name" type="xs:ID"     use="required"/>
+      <xs:attribute name="lib"  type="xs:string" use="optional"/>


What is this for?

I don' think it's used anywhere. This came from the suite_v1.0 schema. I'll comb through the code and if we don't need it, I'll remove it. My guess is that it's from the old days of the "dynamic CCPP".

Removed in 4ac80e4

scripts/ccpp_suite.py

scripts/parse_tools/xml_tools.py

test/nested_suite_test/ccpp_kinds.F90

gold2718 · 2025-11-19T19:11:50Z

Is there a purpose to this branch?

climbfuji · 2025-11-19T19:36:01Z

Is there a purpose to this branch?

No, I must have accidentally pushed to the wrong remote. I deleted it.

gold2718 · 2025-11-19T20:06:14Z

SIMA

@peverwhee. if my new tests get accepted, there are examples of generating the expanded suites. Something like:

        _, xml_root = read_xml_file(suite_source, logger)
        expand_nested_suites(xml_root, <suites dir>, logger=logger)
        write_xml_file(xml_root, expanded_suite, logger)

Can the CAM-SIMA build system just wrap this in a function if it needs an expanded suite file? All the routines are in xml_tools.py

climbfuji · 2025-11-20T03:23:11Z

Interesting, after merging @gold2718's PR into my branch (all tests passed for that PR in my fork), I am now seeing this (https://github.com/NCAR/ccpp-framework/actions/runs/19524388803/job/55894181408):

-- ccpp-capgen completed successfully
-- Running ccpp_datafile from /home/runner/work/ccpp-framework/ccpp-framework/test/advection_test
'/home/runner/work/ccpp-framework/ccpp-framework/scripts/ccpp_datafile.py' '--ccpp-files' '/home/runner/work/ccpp-framework/ccpp-framework/build/test/advection_test/ccpp/datatable.xml'
-- CCPP_CAPS = /home/runner/work/ccpp-framework/ccpp-framework/build/test/advection_test/ccpp/ccpp_kinds.F90,/home/runner/work/ccpp-framework/ccpp-framework/src/ccpp_constituent_prop_mod.F90,/home/runner/work/ccpp-framework/ccpp-framework/src/ccpp_scheme_utils.F90,/home/runner/work/ccpp-framework/ccpp-framework/src/ccpp_hashable.F90,/home/runner/work/ccpp-framework/ccpp-framework/src/ccpp_hash_table.F90,/home/runner/work/ccpp-framework/ccpp-framework/build/test/advection_test/ccpp/test_host_ccpp_cap.F90,/home/runner/work/ccpp-framework/ccpp-framework/build/test/advection_test/ccpp/ccpp_cld_suite_cap.F90
-- CCPP cap files retrieved
-- Creating output directory: /home/runner/work/ccpp-framework/ccpp-framework/build/test/capgen_test/ccpp
-- Running ccpp_capgen.py from /home/runner/work/ccpp-framework/ccpp-framework/test/capgen_test
'/home/runner/work/ccpp-framework/ccpp-framework/scripts/ccpp_capgen.py' '--debug' '--host-files' 'test_host_data.meta,test_host_mod.meta,test_host.meta' '--scheme-files' 'setup_coeffs.meta,temp_set.meta,temp_adjust.meta,temp_calc_adjust.meta,make_ddt.meta,environ_conditions.meta' '--suites' 'ddt_suite.xml,temp_suite.xml' '--host-name' 'test_host' '--output-root' '/home/runner/work/ccpp-framework/ccpp-framework/build/test/capgen_test/ccpp'
CMake Error at cmake/ccpp_capgen.cmake:80 (message):
  CCPP cap generation FAILED: result = 1
Call Stack (most recent call first):
  test/capgen_test/CMakeLists.txt:51 (ccpp_capgen)


-- ccpp-capgen stdout: b'/home/runner/work/ccpp-framework/ccpp-framework/test/capgen_test/ddt_suite.xml validates\n'
b'/home/runner/work/ccpp-framework/ccpp-framework/test/capgen_test/temp_suite.xml validates\n'
Traceback (most recent call last):
  File "/home/runner/work/ccpp-framework/ccpp-framework/scripts/metavar.py", line 1011, in conditional
    int(item)
ValueError: invalid literal for int() with base 10: '('

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/work/ccpp-framework/ccpp-framework/scripts/ccpp_capgen.py", line 752, in <module>
    _main_func()
  File "/home/runner/work/ccpp-framework/ccpp-framework/scripts/ccpp_capgen.py", line 745, in _main_func
    _ = capgen(framework_env)
  File "/home/runner/work/ccpp-framework/ccpp-framework/scripts/ccpp_capgen.py", line 700, in capgen
    ccpp_api = API(sdfs, host_model, scheme_headers, run_env)
  File "/home/runner/work/ccpp-framework/ccpp-framework/scripts/ccpp_suite.py", line 698, in __init__
    suite.analyze(self.host_model, scheme_library,
  File "/home/runner/work/ccpp-framework/ccpp-framework/scripts/ccpp_suite.py", line 460, in analyze
    item.analyze(phase, self, scheme_library, ddt_library,
  File "/home/runner/work/ccpp-framework/ccpp-framework/scripts/suite_objects.py", line 2364, in analyze
    lschemes = item.analyze(phase, self, scheme_library, suite_vars, 1)
  File "/home/runner/work/ccpp-framework/ccpp-framework/scripts/suite_objects.py", line 2130, in analyze
    smods = item.analyze(phase, group, scheme_library,
  File "/home/runner/work/ccpp-framework/ccpp-framework/scripts/suite_objects.py", line 1292, in analyze
    scheme_mods = self.parent.analyze(phase, group, scheme_library,
  File "/home/runner/work/ccpp-framework/ccpp-framework/scripts/suite_objects.py", line 2014, in analyze
    smods = item.analyze(phase, group, scheme_library,
  File "/home/runner/work/ccpp-framework/ccpp-framework/scripts/suite_objects.py", line 1215, in analyze
    self.add_var_debug_check(dict_var)
  File "/home/runner/work/ccpp-framework/ccpp-framework/scripts/suite_objects.py", line 1324, in add_var_debug_check
    (_, vars_needed) = var.conditional(var_dicts)
  File "/home/runner/work/ccpp-framework/ccpp-framework/scripts/metavar.py", line 1020, in conditional
    raise Exception(f"Cannot find variable '{item}' for generating conditional for '{active}'")
Exception: Cannot find variable '(' for generating conditional for '(index_of_water_vapor_specific_humidity > 0)'

-- Configuring incomplete, errors occurred!

mwaxmonsky · 2025-11-20T15:54:46Z

@climbfuji You could modify the ccpp_capgen call in the cmake for the capgen test verbosity level explicity (VERBOSITY ${CCPP_VERBOSITY} -> VERBOSITY 1) and we should be able to see the error message coming from the capgen scripts.

climbfuji · 2025-11-20T22:26:49Z

@climbfuji You could modify the ccpp_capgen call in the cmake for the capgen test verbosity level explicity (VERBOSITY ${CCPP_VERBOSITY} -> VERBOSITY 1) and we should be able to see the error message coming from the capgen scripts.

Thanks. I was able to reproduce this locally, too. But I ended up backing out the PR from @gold2718 entirely, because I am running out of time before the Thanksgiving break and I really want to get this in. Because I force-pushed the older commit, @gold2718 can simply direct his branch/PR to NCAR develop after my PR went in and there won't be any merge conflicts etc. Besides, the changes in @gold2718 didn't really have anything to do with my changes, therefore it's better to keep them separate anyway.

I will address the other questions/issue directly related to my PR before requesting a final review from @gold2718.

…when maximum number of iterations is exceeded

climbfuji added 6 commits October 7, 2025 10:23

Add capability to parse nested suites using a new version of the XML …

4c04333

…schema

Add nested_suite_test

bb5d19e

Add doctests to scripts/parse_tools/xml_tools.py, clean up scripts/cc…

26bba86

…pp_suite.py, scripts/parse_tools/xml_tools.py, and test/nested_suite_test/test_host.F90

More cleanups

7821cef

Merge branch 'develop' of https://github.com/ncar/ccpp-framework into…

559c5e7

… feature/nested_suites

Remove unused test/nested_suite_test/nested_suite_test_reports.py

648612a

climbfuji changed the title ~~Work in progress: Nested suites in capgen~~ Add capability to parse nested suites in capgen Oct 7, 2025

climbfuji marked this pull request as ready for review October 7, 2025 20:24

climbfuji requested review from dustinswales, gold2718, grantfirl, mkavulich, mwaxmonsky and peverwhee as code owners October 7, 2025 20:24

climbfuji added this to capgen unification Oct 7, 2025

climbfuji moved this to In progress in capgen unification Oct 7, 2025

climbfuji self-assigned this Oct 7, 2025

climbfuji added the capgen-unification Issues/PRs necessary for capgen/prebuild unification label Oct 7, 2025

climbfuji linked an issue Oct 7, 2025 that may be closed by this pull request

Add suite keyword to suite definition file? #275

Open

mkavulich reviewed Oct 22, 2025

View reviewed changes

Merge branch 'develop' into feature/nested_suites

219f41b

grantfirl approved these changes Oct 30, 2025

View reviewed changes

peverwhee requested changes Nov 3, 2025

View reviewed changes

scripts/ccpp_suite.py Show resolved Hide resolved

scripts/parse_tools/xml_tools.py Show resolved Hide resolved

climbfuji added 3 commits November 4, 2025 10:45

Merge branch 'develop' of https://github.com/ncar/ccpp-framework into…

a500b4e

… feature/nested_suites

Nested suites: prevent infinite recursion, allow file attribute to be…

0d144ae

… absolute or relative, always write expanded SDF

Merge branch 'feature/nested_suites' of https://github.com/climbfuji/…

57acb17

…ccpp-framework into feature/nested_suites

climbfuji requested review from mkavulich and peverwhee November 4, 2025 19:05

mkavulich approved these changes Nov 4, 2025

View reviewed changes

Merge branch 'develop' into feature/nested_suites

21623ce

Add capability to insert only a specific group of a nested suite at t…

f336c17

…he suite level

Revert back to only one suite per SDF; fix check for recursive expans…

e0d375b

…ion of nested suites

mkavulich approved these changes Nov 15, 2025

View reviewed changes

peverwhee approved these changes Nov 18, 2025

View reviewed changes

peverwhee mentioned this pull request Nov 18, 2025

Create offline way to generate expanded SDF #694

Closed

gold2718 requested changes Nov 19, 2025

View reviewed changes

climbfuji force-pushed the feature/nested_suites branch from db06414 to e0d375b Compare November 20, 2025 22:20

climbfuji added 2 commits November 20, 2025 16:45

In xml_tools.py: collect names of expanded suite and provide to user …

2877943

…when maximum number of iterations is exceeded

Remove unused optional attribute lib from schema/suite_v2_0.xsd

4ac80e4

climbfuji requested a review from gold2718 November 20, 2025 23:33

Add capability to parse nested suites in capgen #691

Are you sure you want to change the base?

Add capability to parse nested suites in capgen #691

Uh oh!

Conversation

climbfuji commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues

Testing

Additional information

Uh oh!

mkavulich left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

grantfirl left a comment

Choose a reason for hiding this comment

Uh oh!

peverwhee left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gold2718 commented Nov 10, 2025

Uh oh!

gold2718 commented Nov 10, 2025

Uh oh!

climbfuji commented Nov 10, 2025

Uh oh!

gold2718 commented Nov 11, 2025

Uh oh!

climbfuji commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

climbfuji commented Nov 12, 2025

Uh oh!

gold2718 commented Nov 12, 2025

Uh oh!

peverwhee commented Nov 13, 2025

Uh oh!

climbfuji commented Nov 13, 2025

Uh oh!

mkavulich commented Nov 14, 2025

Uh oh!

climbfuji commented Nov 14, 2025

Uh oh!

mkavulich left a comment

Choose a reason for hiding this comment

Uh oh!

peverwhee left a comment

Choose a reason for hiding this comment

Uh oh!

gold2718 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

climbfuji commented Oct 7, 2025 •

edited

Loading

climbfuji commented Nov 12, 2025 •

edited

Loading

climbfuji commented Nov 20, 2025 •

edited

Loading