r/Python • u/RomfordNavy • 18d ago
Discussion Alternative to libCST
Looking for a suggestion of an alternative CST parser to libcst. Have been trying to install libcst for over two days now but still unable to do so, I think the issue might be that there is no PyYaml `wheel` for Python 3.15.
So now wondering if there might be some other alternative CST parser available. I need to parse comments as well as other code so standard AST won't work as far as I know.
1
u/RomfordNavy 17d ago
Thanks for the suggestion u/michallandry62 but looks like Parso doesn't support 3.15 yet.
u/Wurstinator thanks for proving that libcast wouild install if I had a C compiler instaled. Was just about to install that but before I did was now considering whether libCST is actually the correct tool for what I want to do.
u/AleDev99 It's primary job is to modify source code and as part of that process one of it's tasks is to replace a holder, say '#replaceThis' with a block of code. I am now trying to establish whether libCST will allow that sort of replacement or whether that needs to be done a separate parse first.
2
u/Nanotheslayer 12d ago
If the markers are standalone comment lines in otherwise valid Python, I'd build on the tokenize suggestion: identify COMMENT tokens, then replace the corresponding source lines. That avoids treating '#replaceThis' inside a triple-quoted string as a placeholder. Keep the marker line's indentation when inserting the block, and apply multiple edits from the bottom up so earlier coordinates stay valid. I'd check a marker inside a string, one in a nested block, and one at EOF before wiring it into the rest of your edits. LibCST still makes sense for the structural changes beyond that.
1
u/AleDev99 17d ago
Yes, libCST can do it in one pass, no separate parse needed. A comment on its own line is kept as an EmptyLine in the leading_lines of the next statement, so a CSTTransformer can catch it there. In leave_SimpleStatementLine / leave_Compound statements (or a generic on_leave for any statement node), check whether any leading_lines comment equals "#replaceThis"; if so, drop that line and return cst.FlattenSentinel([*cst.parse_module(snippet).body, updated_node]), so your new block lands right before that statement with the indentation handled for you. One catch: if the placeholder is the last thing in a block, there's no next statement, so it ends up in the block's footer instead. Honestly, if the placeholder is always on its own line, a plain text replace that keeps the line's indentation is the simplest fix, and you only need libCST for the rest of the edits.
3
u/Wurstinator 18d ago
❯ docker run python:3.15.2 pip install libcst
Collecting libcst
Downloading libcst-1.9.0-cp315-cp315-manylinux_2_28_x86_64.whl.metadata (15 kB)
Collecting pyyaml>=6.0.3 (from libcst)
Downloading pyyaml-6.0.3.tar.gz (130 kB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Downloading libcst-1.9.0-cp315-cp315-manylinux_2_28_x86_64.whl (2.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 10.4 MB/s 0:00:00
Building wheels for collected packages: pyyaml
Building wheel for pyyaml (pyproject.toml): started
Building wheel for pyyaml (pyproject.toml): finished with status 'done'
Created wheel for pyyaml: filename=pyyaml-6.0.3-cp315-cp315-linux_x86_64.whl size=664360 sha256=415311d27ec6094e66e922022c66786c1a8e587c6b944acf466646aaa44c96f3
Stored in directory: /root/.cache/pip/wheels/39/c5/03/188d3f8dc792f835b1e767b5e6e0fec41846dc06f6dd368652
Successfully built pyyaml
Installing collected packages: pyyaml, libcst
Successfully installed libcst-1.9.0 pyyaml-6.0.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.What's the issue?
3
u/michallandry62 18d ago
parso might be what you want. It's the parser jedi uses, it's pure Python so there's nothing to compile, and it keeps comments and whitespace in each node's prefix, so you can round-trip a file back to the exact source.
It's lower level than libcst though, no visitor/transformer layer, so you end up walking the tree yourself. Since your code uses lazy imports, check that its grammar handles that syntax before you commit to it, it ships separate grammar files per Python version.
2
u/AleDev99 17d ago
Wurstinator's log shows libcst itself already ships a cp315 wheel, it's only PyYAML that gets built from source, so the failure is most likely in that build step. Run pip install -v libcst and look at the real error: usually it's a missing C compiler or missing Python 3.15 headers (python3.15-dev or similar on Linux). If you only need parsing, you could also try pip install --no-deps libcst, since as far as I know PyYAML is only used by the codemod tooling, not by libcst.parse_module. On alternatives: parso keeps comments and round-trips source, but it ships a grammar per Python version, so I wouldn't count on it understanding lazy import yet. Worst case, stdlib tokenize gives you every COMMENT token with line/col, which you can match to ast nodes by position.
1
u/danielmartinez22 13d ago
parso round trips source with comments intact and ships a py3 none any wheel so no build step, 3.15 isnt in its classifiers yet so new syntax might trip it
24
u/V0idL0rd 18d ago
Python 3.15 came out very recently, I think a lot of popular libraries are still in the process of being updated, maybe try with python 3.14 for now and upgrade later?