JOJ/JOJ3
24
5
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: more on parsers

manuel 2024-10-17 00:06:00 +08:00
parent 0e78f94858
commit 4e4a1c3b06
1 changed files with 89 additions and 42 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

131
JOJ3-Configuration-Schema.md

@ -48,10 +48,17 @@ Refer to [TOML reference guide](https://toml.io/en/v1.0.0). After writing a conf
Converting the file into JSON format can help better visualize the structure which can be especially helpful when working with arrays of tables.
All JOJ3 configuration files can be found in `course-joj` repository under `home/tt/.config/joj`.
For each repository that will be used in the course (eg. `hw`, `p1`, and `p2`), create a
corresponding folder. This directory will be the `joj-root` for that repository, eg.
`home/tt/.config/joj/hw` is the `joj-root` for the `hw` repositories, ie. where JOJ3 configurations
for homework tasks will be setup.
## Repository level configuration
The first and most simple file to write is `repo.toml`. The template below can be used as a starter.
It contains the part of the configuration that will be used globally for all assignments and tasks.
It should be saved in the `joj-root` configuration for the repository.
- `teaching_team [array of string]`: TT members' jaccount
- `max_size [float]`: maximum size allowed for a repo in MB
@ -70,13 +77,13 @@ It contains the part of the configuration that will be used globally for all ass
- put this `repo.toml` file in the "root directory" containing all the configuration for this
repository, eg. `/home/tt/.config/joj/hw` for homework repository
<details><summary>Sample repo.toml</summary>
<details><summary>Sample repo.toml -- stored in the `joj-root` configuration for the repository</summary>
```toml
teaching_team = ["mac-wang", "jon-lee", "allen_wr"] # jaccounts
max_size = 5 # 5MB repo max size
release_tags = ["h1", "h2", "h3"] # list of valid release tags
maxsize = 5 # 5MB repo max size
releasetags = ["h1", "h2", "h3"] # list of valid release tags
[files]
immutable = [".gitignore", ".gitattributes", ".gitea/workflows/push.yaml"] # readonly files
@ -92,34 +99,47 @@ This configuration file will be used to generate the task level configuration of
should therefore clearly describe what stages to run, how to run them, and what information to
share back with students.
### General options
Global setup:
Global options:
- `task [string]`: name the task (eg. an exercise or project milestone)
- `release.stages [array of string]`: list all stages to run on a release
- `release.stages [array of string]`: list all stages to run on a release (default: `[ ]`)
- `release.deadline [offset date-time]`: RFC 3339 formatted date-time with offset
- `limit.cpu [int]`: default maximum running time used for all stages in sec (default: `4`)
- `limit.mem [int]`: default maximum amount of RAM allowed for all stages in MB (default: `4`)
Each stage is configured using a table. All parameters following a table definition belong to it until
Each stage is configured in a table. All parameters following a table definition belong to it until
the next table is defined.
`[stage1]` table: configuration for stage `stage1`
`[stagename]` table: configuration for stage `stagename`
- `command [string]`: command to run
- `files.import [array of string]`: list of files to import to ensure the command runs as expected (eg.
driver and header files needed for compilation)
- `files.export [array of string]`: list of generated files to export to ensure future commands run as expected (eg.
binaries needed for online-judge stages)
- `parsers [array of string]`: list of parsers to run on the result of command
- `limit.cpu [int]`: maximum running time for the stage or step in sec
- `limit.mem [int]`: maximum amount of RAM allowed for stage of step in MB
-
- `parsers [array of string]`: list of parsers to run on the output of `command`
- `limit.cpu [int]`: maximum running time for the stage in sec (default: `4`)
- `limit.mem [int]`: maximum amount of RAM allowed for stage of step in MB (default: `4`)
Any online-judge stage *must* feature the keyword `judge`, eg. `judge`, `judge-base`, `asan-judge` are all valid online-judge
stage names while `oj`, `run-base`, and `asan` are not.
An online-judge stage is configure as any other stage, ie. using the above options, but can feature an
extra parameter:
- `skip [array of string]`: list of test cases to skip for this stage (default: `[ ]`)
By default all available test-cases are tested, unless some are marked as `skip`.
### Parsers
Currently the following parsers are available:
- Generic:
- `dummy`: output the score and comment
- `keyword`: catch keywords on any generic text output and can force quit on a match
- `result-detail`: provide basic statistics on memory and CPU usage
- `result-status`: check if the executor exit with status accepted, if not quit with error status
- `dummy`: output the score and comment in the settings (only use for testing purpose)
- Code quality:
- `clangtidy`: parse clang-tidy output for specified keywords
- `cppcheck`: parse cppcheck output for specified keywords
@ -128,40 +148,67 @@ Currently the following parsers are available:
- Online judge
- `diff`: difference between the output and a provided file content (commonly used for judge stage)
diff
- `comment.pass`
- `comment.fail`
- `score`
- `output.hide`
- `output.ignorespaces`
- `forcequit`
Parsers can be combined. For instance one might want to show the `diff`, `result-status`, and `result-detail`
outputs for the online judge. Some parsers can also be further configured.
dummy
- `comment`
- `score`
clangtidy cppcheck cpplint
- `keyword`
- `weight`
#### Dummy parser options
keyword
- quitonmatch
- `comment [string]`: display any text
- `score [int]`: score for passing dummy stage (default: `0`)
status
- comment
- score
*Note.* Adding a dummy stage can be useful to add extra comments or separate parsers output.
details
- time
- mem
- stderr
- stdout
- exit status
#### Keyword parser
Note that parsers can be combined. For instance one might want to show the `diff`, `result-status`, and `result-detail`
outputs for the online judge.
- `forcequit [boolean]`: quit at the end of this stage if there is a match
- `keyword [array of string]`: list of keywords to catch on `stdout`
- `weight [array of int]`: corresponding weight for each keyword caught on `stdout`
*Note.* This parser can be used to catch words on the output of any code quality tool which
doesn't already have a parser implement in JOJ3
#### Result-detail parser
- `exitstatus [boolean]`: display exit status (default: `false`)
- `mem [boolean]`: display the memory usage (default: `true`)
- `stderr [boolean]`: display `stderr` messages (default: `false`)
- `stdout [boolean]`: display `stdout` messages (default: `false`)
- `time [boolean]`: display run time (default: `true`)
#### Result-status parser
- `comment [string]`: comment to display on successful run
- `score [int]`: score to assign on a successful run (default: `0`)
*Note.* The main benefit of this parser is that it quits on failure. It could be used to exit at
the end of a failed compilation stage.
#### Clangtidy, cppcheck, and cpplint parsers
- `keyword [array of string]`: list of keywords to catch on `stdout`
- `weight [array of int]`: corresponding weight for each keyword caught on `stdout`
*Warning.* Current implementation of cpplint parser is incomplete, so weight and keywords might be
ignored; for now cpplint parser catches all cpplint checks and apply scoring automatically.
#### elf parser
Not ready yet.
#### Diff parser options
- `comment.pass [string]`: comment to display when passing a test case (default: `"🥳Passed!"`)
- `comment.fail [string]`: comment to display when failing a test case (default: `"🧐Failed..."`)
- `forcequit [boolean]`: quit OJ on a failed test case (default: `false`
- `output.hide [boolean]`: hide diff output (default: `false`)
- `output.ignorespaces [boolean]`: ignore white spaces in diff output (default: `true`)
- `score [int]`: score awarded for passing the test case (default: `0`)
*Note.* This parser can be configured and adjusted for each test-case (others parsers are configured at the stage level)
Some parsers can also be further configured.
<details><summary>Sample task.toml</summary>
@ -173,8 +220,8 @@ release.deadline = 2024-10-12 23:59:00+08:00
release.stages = [ "compile" ]
[compile]
command = "cmake"
files.import = [ "main.c", "task.h", "CMakelist.txt" ] # files to include with repo code when compiling
command = "make.sh" # eg. script running cmake commands
files.import = [ "src/main.c", "src/task.h", "srcCMakelist.txt" ] #
files.export = [ "p1", "p1-msan" ]
# limit.cpu = 300 # allow 300s for complex/long compilation
@ -192,7 +239,7 @@ weights = [ 50, 20 ] # weight of each keyword
parsers
[clangtidy]
command = "clang-tidy-18 -header-filter=.* -quiet -load=/usr/local/lib/libcodequality.so -p build"
command = "run-clang-tidy-18 -header-filter=.* -quiet -load=/usr/local/lib/libcodequality.so -p build"
tests = [ "codequality-no-global-variables", "codequality-no-header-guard", "readability-function-size", "readability-duplicate-include", "readability-identifier-naming", "readability-redundant", "readability-misleading-indentation", "readability-misplaced-array-index", "cppcoreguidelines-init-variables", "bugprone-suspicious-string-compare", "google-global-names-in-headers", "clang-diagnostic", "clang-analyzer", "misc performance" ]
weights = [100, 100, 50, 10, 5, 5, 10, 5, 5, 8, 5, 5, 5, 5, 8]