Changes in c_lex.py

5181afcdf99527dd92a88f80fc4d0d8013e1b510 - Merge tag 'docs-7.1' of git://git.kernel.org/pub/scm/linux/kernel/git/docs/linux

Tue, 14 Apr 2026 15:47:08 +0000

Merge tag 'docs-7.1' of git://git.kernel.org/pub/scm/linux/kernel/git/docs/linuxPull documentation updates from Jonathan Corbet: "A busier cycle than I had expected for docs, including: - Translations: some overdue updates to the Japanese translations, Chinese translations for some of the Rust documentation, and the beginnings of a Portuguese translation. - New documents covering CPU isolation, managed interrupts, debugging Python gbb scripts, and more. - More tooling work from Mauro, reducing docs-build warnings, adding self tests, improving man-page output, bringing in a proper C tokenizer to replace (some of) the mess of kernel-doc regexes, and more. - Update and synchronize changes.rst and scripts/ver_linux, and put both into alphabetical order. ... and a long list of documentation updates, typo fixes, and general improvements"* tag 'docs-7.1' of git://git.kernel.org/pub/scm/linux/kernel/git/docs/linux: (162 commits) Documentation: core-api: real-time: correct spelling doc: Add CPU Isolation documentation Documentation: Add managed interrupts Documentation: seq_file: drop 2.6 reference docs/zh_CN: update rust/index.rst translation docs/zh_CN: update rust/quick-start.rst translation docs/zh_CN: update rust/coding-guidelines.rst translation docs/zh_CN: update rust/arch-support.rst translation docs/zh_CN: sync process/2.Process.rst with English version docs/zh_CN: fix an inconsistent statement in dev-tools/testing-overview tracing: Documentation: Update histogram-design.rst for fn() handling docs: sysctl: Add documentation for /proc/sys/xen/ Docs: hid: intel-ish-hid: make long URL usable Documentation/kernel-parameters: fix architecture alignment for pt, nopt, and nobypass sched/doc: Update yield_task description in sched-design-CFS Documentation/rtla: Convert links to RST format docs: fix typos and duplicated words across documentation docs: fix typo in zoran driver documentation docs: add an Assisted-by mention to submitting-patches.rst Revert "scripts/checkpatch: add Assisted-by: tag validation" ... List of files: /linux/tools/lib/python/kdoc/c_lex.py

2ca0b54dca438edb0f6b0eec7913d3cab60ddebf - docs: c_lex.py: store logger on its data

Mon, 23 Mar 2026 09:10:53 +0000

docs: c_lex.py: store logger on its dataBy having the logger stored there, any code using CTokenizer canlog messages there.Signed-off-by: Mauro Carvalho Chehab Signed-off-by: Jonathan Corbet Message-ID: <467979dc18149e4b2a7113c178e0cb07919632f2.1774256269.git.mchehab+huawei@kernel.org> List of files: /linux/tools/lib/python/kdoc/c_lex.py

0a1a27776ddf0072883cdb4a61b91155553fcb96 - Merge branch 'mauro' into docs-mw

Sun, 22 Mar 2026 21:25:08 +0000

Merge branch 'mauro' into docs-mwThis series comes after: https://lore.kernel.org/linux-doc/cover.1773770483.git.mchehab+huawei@kernel.org/It basically contains patches I submitted before on a 40+ patch series,but were less relevant, plus a couple of other minor fixes:- patch 1 improves one of the CTokenizer unit test, fixing some potential issues on it;- patches 2 and 3 contain some improvement/fixes for Sphinx Python autodoc extension. They basically document c_lex.py;- The remaining patches: - create a new class for kernel-doc config; - fix some internal representations of KdocItem; - add unit tests for KernelDoc() parser class; - add support to output KdocItem in YAML, which is a machine-readable output for all documented kAPI.None of the patches should affect man or html output. List of files: /linux/tools/lib/python/kdoc/c_lex.py

8c0b7c0d3c0e640b3ebb7f1f648ea322e56c227a - docs: kdoc: add c_lex to generated documentation

Wed, 18 Mar 2026 09:11:03 +0000

docs: kdoc: add c_lex to generated documentationDo some fixes at groups() description for it to be parsed bySphinx and add it to the documentation.Signed-off-by: Mauro Carvalho Chehab Signed-off-by: Jonathan Corbet Message-ID: <799178cf30dd4022fdb1d029ba998a458e037b52.1773823995.git.mchehab+huawei@kernel.org> List of files: /linux/tools/lib/python/kdoc/c_lex.py

781171bec0650c00c642564afcb5cce57abda5bf - Merge branch 'mauro' into docs-mw

Sun, 22 Mar 2026 21:06:59 +0000

Merge branch 'mauro' into docs-mwMauro says:This patch series change how kdoc parser handles macro replacements.Instead of heavily relying on regular expressions that can sometimesbe very complex, it uses a C lexical tokenizer. This ensures thatBEGIN/END blocks on functions and structs are properly handled,even when nested.Checking before/after the patch series, for both man pages andrst only had: - whitespace differences; - struct_group macros now are shown as inner anonimous structs as it should be.Also, I didn't notice any relevant change on the documentation buildtime. With that regards, right now, every time a CMatch replacementrule takes in place, it does: for each transform: - tokenizes the source code; - handle CMatch; - convert tokens back to a string.A possible optimization would be to do, instead: - tokenizes source code; - for each transform handle CMatch; - convert tokens back to a string.For now, I opted not do do it, because: - too much changes on a single row; - docs build time is taking ~3:30 minutes, which is about the same time it ws taken before the changes; - there is a very dirty hack inside function_xforms: (KernRe(r"_noprof"), ""). This is meant to change function prototypes instead of function arguments.So, if ok for you, I would prefer to merge this one first. We can lateroptimize kdoc_parser to avoid multiple token <-> string conversions.-One important aspect of this series is that it introduces unittestsfor kernel-doc. I used it a lot during the development of this series,to ensure that the changes I was doing were producing the expectedresults. Tests are on two separate files that can be executed directly.Alternatively, there is a run.py script that runs all of them (andany other python script named tools/unittests/test_*.py"): $ tools/unittests/run.py test_cmatch: TestSearch: test_search_acquires_multiple: OK test_search_acquires_nested_paren: OK test_search_acquires_simple: OK test_search_must_hold: OK test_search_must_hold_shared: OK test_search_no_false_positive: OK test_search_no_function: OK test_search_no_macro_remains: OK TestSubMultipleMacros: test_acquires_multiple: OK test_acquires_nested_paren: OK test_acquires_simple: OK test_mixed_macros: OK test_must_hold: OK test_must_hold_shared: OK test_no_false_positive: OK test_no_function: OK test_no_macro_remains: OK TestSubSimple: test_rise_early_greedy: OK test_rise_multiple_greedy: OK test_strip_multiple_acquires: OK test_sub_count_parameter: OK test_sub_mixed_placeholders: OK test_sub_multiple_placeholders: OK test_sub_no_placeholder: OK test_sub_single_placeholder: OK test_sub_with_capture: OK test_sub_zero_placeholder: OK TestSubWithLocalXforms: test_functions_with_acquires_and_releases: OK test_raw_struct_group: OK test_raw_struct_group_tagged: OK test_struct_group: OK test_struct_group_attr: OK test_struct_group_tagged_with_private: OK test_struct_kcov: OK test_vars_stackdepot: OK test_tokenizer: TestPublicPrivate: test_balanced_inner_private: OK test_balanced_non_greddy_private: OK test_balanced_private: OK test_no private: OK test_unbalanced_inner_private: OK test_unbalanced_private: OK test_unbalanced_struct_group_tagged_with_private: OK test_unbalanced_two_struct_group_tagged_first_with_private: OK test_unbalanced_without_end_of_line: OK TestTokenizer: test_basic_tokens: OK test_depth_counters: OK test_mismatch_error: OK Ran 47 tests List of files: /linux/tools/lib/python/kdoc/c_lex.py

024e200e2a89d71dceff7d1bff8ae77b145726e0 - docs: c_lex: setup a logger to report tokenizer issues

Tue, 17 Mar 2026 18:09:38 +0000

docs: c_lex: setup a logger to report tokenizer issuesReport file that has issues detected via CMatch and CTokenizer.This is done by setting up a logger that will be overriden bykdoc_parser, when used on it.Signed-off-by: Mauro Carvalho Chehab Reviewed-by: Aleksandr Loktionov Signed-off-by: Jonathan Corbet Message-ID: <903ad83ae176196a50444e66177a4f5bcdef5199.1773770483.git.mchehab+huawei@kernel.org> List of files: /linux/tools/lib/python/kdoc/c_lex.py

9aaeb817ef4f794d1dbb8736332a64b5dae9521c - docs: c_lex: properly implement a sub() method for CMatch

Tue, 17 Mar 2026 18:09:32 +0000

docs: c_lex: properly implement a sub() method for CMatchImplement a sub() method to do what it is expected, parsingbackref arguments like \0, \1, \2, ...Signed-off-by: Mauro Carvalho Chehab Signed-off-by: Jonathan Corbet Message-ID: List of files: /linux/tools/lib/python/kdoc/c_lex.py

f1cf9f7cd66f1f90c4c3beb0885b6f7771e1b419 - docs: kdoc: create a CMatch to match nested C blocks

Tue, 17 Mar 2026 18:09:30 +0000

docs: kdoc: create a CMatch to match nested C blocksThe NextMatch code is complex, and will become even more complexif we add there support for arguments.Now that we have a tokenizer, we can use a better solution,easier to be understood.Yet, to improve performance, it is better to make it use apreviously tokenized code, changing its ABI.So, reimplement NextMatch using the CTokener class. Once itis done, we can drop NestedMatch.Signed-off-by: Mauro Carvalho Chehab Signed-off-by: Jonathan Corbet Message-ID: List of files: /linux/tools/lib/python/kdoc/c_lex.py

df50e848f67523195ee0b4c6d2c01823e36a15e7 - docs: add a C tokenizer to be used by kernel-doc

Tue, 17 Mar 2026 18:09:25 +0000

docs: add a C tokenizer to be used by kernel-docHandling C code purely using regular expressions doesn't work well.Add a C tokenizer to help doing it the right way.The tokenizer was written using as basis the Python re documentationtokenizer example from: https://docs.python.org/3/library/re.html#writing-a-tokenizerSigned-off-by: Mauro Carvalho Chehab Signed-off-by: Jonathan Corbet Message-ID: <39787bb8022e10c65df40c746077f7f66d07ffed.1773770483.git.mchehab+huawei@kernel.org> List of files: /linux/tools/lib/python/kdoc/c_lex.py