r/Python • • 18d ago

Discussion Alternative to libCST

Looking for a suggestion of an alternative CST parser to libcst. Have been trying to install libcst for over two days now but still unable to do so, I think the issue might be that there is no PyYaml `wheel` for Python 3.15.

So now wondering if there might be some other alternative CST parser available. I need to parse comments as well as other code so standard AST won't work as far as I know.

15 Upvotes

14 comments sorted by

24

u/V0idL0rd 18d ago

Python 3.15 came out very recently, I think a lot of popular libraries are still in the process of being updated, maybe try with python 3.14 for now and upgrade later?

-3

u/RomfordNavy 18d ago

Not so easy to do in this case as code relies on lazy imports which didn't exist in 3.14.

9

u/coderanger 18d ago

So remove them? It's literally a 2 week old feature, have you made so many changes in that time as to make backing them out impossible?

0

u/RomfordNavy 18d ago

Been working with lazy imports for a few months now on earlier non-final releases of 3.15. In order to avoid cyclical imports much of the code relies upon lazy imports, really don't want to have to revert to the previous way I was doing things as much has changed since then.

5

u/really_not_unreal 18d ago

In that case you'll have to live without.

7

u/mr_frpdo 18d ago

Technically Lazy is just synthetic sugar. It can be rewritten to use with out it.

1

u/RomfordNavy 17d ago

Thanks for the suggestion u/michallandry62 but looks like Parso doesn't support 3.15 yet.

u/Wurstinator thanks for proving that libcast wouild install if I had a C compiler instaled. Was just about to install that but before I did was now considering whether libCST is actually the correct tool for what I want to do.

u/AleDev99 It's primary job is to modify source code and as part of that process one of it's tasks is to replace a holder, say '#replaceThis' with a block of code. I am now trying to establish whether libCST will allow that sort of replacement or whether that needs to be done a separate parse first.

2

u/Nanotheslayer 12d ago

If the markers are standalone comment lines in otherwise valid Python, I'd build on the tokenize suggestion: identify COMMENT tokens, then replace the corresponding source lines. That avoids treating '#replaceThis' inside a triple-quoted string as a placeholder. Keep the marker line's indentation when inserting the block, and apply multiple edits from the bottom up so earlier coordinates stay valid. I'd check a marker inside a string, one in a nested block, and one at EOF before wiring it into the rest of your edits. LibCST still makes sense for the structural changes beyond that.

1

u/AleDev99 17d ago

Yes, libCST can do it in one pass, no separate parse needed. A comment on its own line is kept as an EmptyLine in the leading_lines of the next statement, so a CSTTransformer can catch it there. In leave_SimpleStatementLine / leave_Compound statements (or a generic on_leave for any statement node), check whether any leading_lines comment equals "#replaceThis"; if so, drop that line and return cst.FlattenSentinel([*cst.parse_module(snippet).body, updated_node]), so your new block lands right before that statement with the indentation handled for you. One catch: if the placeholder is the last thing in a block, there's no next statement, so it ends up in the block's footer instead. Honestly, if the placeholder is always on its own line, a plain text replace that keeps the line's indentation is the simplest fix, and you only need libCST for the rest of the edits.

3

u/Wurstinator 18d ago
❯ docker run python:3.15.2 pip install libcst
Collecting libcst
  Downloading libcst-1.9.0-cp315-cp315-manylinux_2_28_x86_64.whl.metadata (15 kB)
Collecting pyyaml>=6.0.3 (from libcst)
  Downloading pyyaml-6.0.3.tar.gz (130 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Downloading libcst-1.9.0-cp315-cp315-manylinux_2_28_x86_64.whl (2.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 10.4 MB/s  0:00:00
Building wheels for collected packages: pyyaml
  Building wheel for pyyaml (pyproject.toml): started
  Building wheel for pyyaml (pyproject.toml): finished with status 'done'
  Created wheel for pyyaml: filename=pyyaml-6.0.3-cp315-cp315-linux_x86_64.whl size=664360 sha256=415311d27ec6094e66e922022c66786c1a8e587c6b944acf466646aaa44c96f3
  Stored in directory: /root/.cache/pip/wheels/39/c5/03/188d3f8dc792f835b1e767b5e6e0fec41846dc06f6dd368652
Successfully built pyyaml
Installing collected packages: pyyaml, libcst

Successfully installed libcst-1.9.0 pyyaml-6.0.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.

What's the issue?

3

u/wRAR_ 17d ago

Building wheels for collected packages: pyyaml

What's the issue?

They may be using a platform without a (n easily installable) compiler.

3

u/michallandry62 18d ago

parso might be what you want. It's the parser jedi uses, it's pure Python so there's nothing to compile, and it keeps comments and whitespace in each node's prefix, so you can round-trip a file back to the exact source.

It's lower level than libcst though, no visitor/transformer layer, so you end up walking the tree yourself. Since your code uses lazy imports, check that its grammar handles that syntax before you commit to it, it ships separate grammar files per Python version.

2

u/AleDev99 17d ago

Wurstinator's log shows libcst itself already ships a cp315 wheel, it's only PyYAML that gets built from source, so the failure is most likely in that build step. Run pip install -v libcst and look at the real error: usually it's a missing C compiler or missing Python 3.15 headers (python3.15-dev or similar on Linux). If you only need parsing, you could also try pip install --no-deps libcst, since as far as I know PyYAML is only used by the codemod tooling, not by libcst.parse_module. On alternatives: parso keeps comments and round-trips source, but it ships a grammar per Python version, so I wouldn't count on it understanding lazy import yet. Worst case, stdlib tokenize gives you every COMMENT token with line/col, which you can match to ast nodes by position.

1

u/danielmartinez22 13d ago

parso round trips source with comments intact and ships a py3 none any wheel so no build step, 3.15 isnt in its classifiers yet so new syntax might trip it