fortls crashes during initialization due to RecursionError

[tomoto]

Describe the bug

fortls crashes during initialization due to RecursionError when the workspace contains source files with deep nesting levels (about 200 nestings of IF~ENDIF). While it is acceptable to skip parsing such files, the bigger issue is fortls goes down entirely without any information to help me troubleshoot.

The output message:

["INFO" - 10:24:53 PM] Fortran Language Server
["INFO" - 10:24:53 PM] Initialising Language Server for workspace: /home/tomoto/dev/fortran/test1.f with command-line options: --enable_code_actions, --hover_signature, --use_signature_help, --nthreads=4, --notify_init, --incremental_sync
[WARN - 22:24:54] error handling request {'jsonrpc': '2.0', 'id': 0, 'method': 'initialize', 'params': {'processId': 1682974, 'clientInfo': {'name': 'Visual Studio Code', 'version': '1.95.3'}, 'locale': 'en', 'rootPath': '/home/tomoto/dev/fortran', 'rootUri': 'file:///home/tomoto/dev/fortran', 'capabilities': {'workspace': {'applyEdit': True, 'workspaceEdit': {'documentChanges': True, 'resourceOperations': ['create', 'rename', 'delete'], 'failureHandling': 'textOnlyTransactional', 'normalizesLineEndings': True, 'changeAnnotationSupport': {'groupsOnLabel': True}}, 'configuration': True, 'didChangeWatchedFiles': {'dynamicRegistration': True, 'relativePatternSupport': True}, 'symbol': {'dynamicRegistration': True, 'symbolKind': {'valueSet': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26]}, 'tagSupport': {'valueSet': [1]}, 'resolveSupport': {'properties': ['location.range']}}, 'codeLens': {'refreshSupport': True}, 'executeCommand': {'dynamicRegistration': True}, 'didChangeConfiguration': {'dynamicRegistration': True}, 'semanticTokens': {'refreshSupport': True}, 'fileOperations': {'dynamicRegistration': True, 'didCreate': True, 'didRename': True, 'didDelete': True, 'willCreate': True, 'willRename': True, 'willDelete': True}, 'inlineValue': {'refreshSupport': True}, 'inlayHint': {'refreshSupport': True}, 'diagnostics': {'refreshSupport': True}}, 'textDocument': {'publishDiagnostics': {'relatedInformation': True, 'versionSupport': False, 'tagSupport': {'valueSet': [1, 2]}, 'codeDescriptionSupport': True, 'dataSupport': True}, 'synchronization': {'dynamicRegistration': True, 'willSave': True, 'willSaveWaitUntil': True, 'didSave': True}, 'completion': {'dynamicRegistration': True, 'contextSupport': True, 'completionItem': {'snippetSupport': True, 'commitCharactersSupport': True, 'documentationFormat': ['markdown', 'plaintext'], 'deprecatedSupport': True, 'preselectSupport': True, 'tagSupport': {'valueSet': [1]}, 'insertReplaceSupport': True, 'resolveSupport': {'properties': ['documentation', 'detail', 'additionalTextEdits']}, 'insertTextModeSupport': {'valueSet': [1, 2]}, 'labelDetailsSupport': True}, 'insertTextMode': 2, 'completionItemKind': {'valueSet': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25]}, 'completionList': {'itemDefaults': ['commitCharacters', 'editRange', 'insertTextFormat', 'insertTextMode']}}, 'hover': {'dynamicRegistration': True, 'contentFormat': ['markdown', 'plaintext']}, 'signatureHelp': {'dynamicRegistration': True, 'signatureInformation': {'documentationFormat': ['markdown', 'plaintext'], 'parameterInformation': {'labelOffsetSupport': True}, 'activeParameterSupport': True}, 'contextSupport': True}, 'definition': {'dynamicRegistration': True, 'linkSupport': True}, 'references': {'dynamicRegistration': True}, 'documentHighlight': {'dynamicRegistration': True}, 'documentSymbol': {'dynamicRegistration': True, 'symbolKind': {'valueSet': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26]}, 'hierarchicalDocumentSymbolSupport': True, 'tagSupport': {'valueSet': [1]}, 'labelSupport': True}, 'codeAction': {'dynamicRegistration': True, 'isPreferredSupport': True, 'disabledSupport': True, 'dataSupport': True, 'resolveSupport': {'properties': ['edit']}, 'codeActionLiteralSupport': {'codeActionKind': {'valueSet': ['', 'quickfix', 'refactor', 'refactor.extract', 'refactor.inline', 'refactor.rewrite', 'source', 'source.organizeImports']}}, 'honorsChangeAnnotations': False}, 'codeLens': {'dynamicRegistration': True}, 'formatting': {'dynamicRegistration': True}, 'rangeFormatting': {'dynamicRegistration': True}, 'onTypeFormatting': {'dynamicRegistration': True}, 'rename': {'dynamicRegistration': True, 'prepareSupport': True, 'prepareSupportDefaultBehavior': 1, 'honorsChangeAnnotations': True}, 'documentLink': {'dynamicRegistration': True, 'tooltipSupport': True}, 'typeDefinition': {'dynamicRegistration': True, 'linkSupport': True}, 'implementation': {'dynamicRegistration': True, 'linkSupport': True}, 'colorProvider': {'dynamicRegistration': True}, 'foldingRange': {'dynamicRegistration': True, 'rangeLimit': 5000, 'lineFoldingOnly': True, 'foldingRangeKind': {'valueSet': ['comment', 'imports', 'region']}, 'foldingRange': {'collapsedText': False}}, 'declaration': {'dynamicRegistration': True, 'linkSupport': True}, 'selectionRange': {'dynamicRegistration': True}, 'callHierarchy': {'dynamicRegistration': True}, 'semanticTokens': {'dynamicRegistration': True, 'tokenTypes': ['namespace', 'type', 'class', 'enum', 'interface', 'struct', 'typeParameter', 'parameter', 'variable', 'property', 'enumMember', 'event', 'function', 'method', 'macro', 'keyword', 'modifier', 'comment', 'string', 'number', 'regexp', 'operator', 'decorator'], 'tokenModifiers': ['declaration', 'definition', 'readonly', 'static', 'deprecated', 'abstract', 'async', 'modification', 'documentation', 'defaultLibrary'], 'formats': ['relative'], 'requests': {'range': True, 'full': {'delta': True}}, 'multilineTokenSupport': False, 'overlappingTokenSupport': False, 'serverCancelSupport': True, 'augmentsSyntaxTokens': True}, 'linkedEditingRange': {'dynamicRegistration': True}, 'typeHierarchy': {'dynamicRegistration': True}, 'inlineValue': {'dynamicRegistration': True}, 'inlayHint': {'dynamicRegistration': True, 'resolveSupport': {'properties': ['tooltip', 'textEdits', 'label.tooltip', 'label.location', 'label.command']}}, 'diagnostic': {'dynamicRegistration': True, 'relatedDocumentSupport': False}}, 'window': {'showMessage': {'messageActionItem': {'additionalPropertiesSupport': True}}, 'showDocument': {'support': True}, 'workDoneProgress': True}, 'general': {'staleRequestSupport': {'cancel': True, 'retryOnContentModified': ['textDocument/semanticTokens/full', 'textDocument/semanticTokens/range', 'textDocument/semanticTokens/full/delta']}, 'regularExpressions': {'engine': 'ECMAScript', 'version': 'ES2020'}, 'markdown': {'parser': 'marked', 'version': '1.1.0'}, 'positionEncodings': ['utf-16']}, 'notebookDocument': {'synchronization': {'dynamicRegistration': True, 'executionSummarySupport': True}}}, 'trace': 'off', 'workspaceFolders': [{'uri': 'file:///home/tomoto/dev/fortran', 'name': 'fortran'}]}}
Traceback (most recent call last):
  File "/home/tomoto/dev/fortls/fortls/langserver.py", line 173, in handle
    resp = handler(request)
  File "/home/tomoto/dev/fortls/fortls/langserver.py", line 213, in serve_initialize
    self.workspace_init()
  File "/home/tomoto/dev/fortls/fortls/langserver.py", line 1492, in workspace_init
    result_obj = result.get()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '<fortls.parsers.internal.parser.FortranFile object at 0x7fc3f96db5e0>'. Reason: 'RecursionError('maximum recursion depth exceeded while pickling an object')'
[Error - 10:24:54 PM] Server initialization failed.
  Message: Error sending result: '<fortls.parsers.internal.parser.FortranFile object at 0x7fc3f96db5e0>'. Reason: 'RecursionError('maximum recursion depth exceeded while pickling an object')'
  Code: -32603 
[object Object]
[Error - 10:24:54 PM] Fortran Language Server client: couldn't create connection to server.
  Message: Error sending result: '<fortls.parsers.internal.parser.FortranFile object at 0x7fc3f96db5e0>'. Reason: 'RecursionError('maximum recursion depth exceeded while pickling an object')'
  Code: -32603 
[object Object]

How it looks like in the VSCode GUI:

As far as I can tell, the RecursionError occurs while pickling (serialization/deserialization) the AST to pass the parse result from the worker process to the main process. It can happen when the AST has a deep nesting of scopes, and possibly for many other reasons.

To Reproduce

1. Place a source file with deep nesting levels in the workspace. Use the Python script below to create such a file:

   levels = 200
   print("      PROGRAM NESTED_IF")
   print("      IF (.TRUE.) THEN\n" * levels, end="")
   print("      END IF\n" * levels, end="")
   print("      END PROGRAM NESTED_IF")

2. Open the workspace and start fortls.
3. Observe "Server initialization failed" error and you cannot use any function of fortls. You see no information about what caused it.

Expected behavior

Fortls should gracefully ignore such files instead of crashing entirely. Besides, user should be informed about which files caused the issue, so they can choose to remove those files from the workspace if desired.

I understand that such deeply nested source files are very rare, possibly machine-generated. I don't necessarily expect fortls to process them, as they may require additional resources and time that may not be worthwhile. I'm fine as long as fortls continues to function without being impacted by such exotic files.

Screenshots & Animations

Nothing to add.

Setup information (please complete the following information):

• OS: Linux
• Python Version: 3.10.12
• fortls Version: 3.1.3
• Code editor used: VS Code
• the Fortran extension for the code editor and its version: Modern Fortran v3.2.0

Configuration information (please complete the following information):

Nothing special.

Additional context

I will post a PR of the suggested fix. Thank you very much!

[tomoto]

@gnikit I posted the PR at #446. Appreciate your time!

[gnikit]

There is a CLI option that extends the Python recursion limit for extremely heavily nested projects see can you give that a go? Probably still a good idea to add your fix but with a more limited catch.

[tomoto]

Thank you for guiding me to the CLI option that I overlooked. I increased --recursion_limit to 10000, and fortls successfully processed the files with deep nestings.

So, my proposed fix is no longer a fix of an "issue", but an improvement of the troubleshooting experience by (1) gracefully ignoring the file that caused the error rather than crashing the entire process and (2) letting user know which file caused the error. I leave the decision to you, and I would be happy to help if you needed anything from my side.

Thanks again for your time to look at this!

P.S. It may be a little bit tricky to reasonably define the exception types to catch. It is because multiprocessing.pool.AsyncResult's get() may not only throw RecursionError as reported here but also rethrow whatever exceptions that occurred in the child process. They are unpredictable, but still can be ignored gracefully since it has no impact on the parent process. So, in this particular case, you may want to set a large net to catch unexpected exceptions and it may be eventually helpful for user to manage any random unexpected scenarios.

[gnikit]

Okay so I looked into this. It's not possible to capture the original exception, since the RecursionError is raised by the pickling action of the multiprocessing.pool. The final exception being raised is multiprocessing.pool.MaybeEncodingError, however the cause (i.e. exc) has been serialised to a str so it's possible to recover the base exception which is RecursionError.

The suggested solution is okay, however the user has no idea what to do with this information. If it wasn't clear to you, who actually took the time to look at the code, I doubt that it will be clear to a novice user that they need to increase the --recursion_limit.
Also, I have no idea if this should be an error in terms of severity, somehow in my tests initialisation still completed and LSP features worked fine.

The only solutions I can think that work are:

1. scan the exception string for RecursionError and post a relevant message to update recursion_limit
2. pickle the AST object which would raise the exception in file_init although I have no idea if that will have negative performance side effects with Pool pickling the object once again.

[tomoto]

Thank you for addressing this! Yes, it would be nice if we could make the information clear enough for a novice user to find the exact solution or workaround easily, but at the same time, we want to avoid over-engineering. I think the final solution strikes a good balance for now.

To me, it's a significant improvement that the user at least understands the issue is caused by specific files rather than the entire project. When I first encountered this issue, I mistakenly thought my project was too large for fortls to handle and ended up giving up on using it for months. I hope this fix will make things easier for other users than it was for me. Thanks again!