Compare commits
3 Commits
0b248aad6b
...
279dce65d1
Author | SHA1 | Date | |
---|---|---|---|
279dce65d1 | |||
753c2d482e | |||
9165e78a16 |
176
README.md
176
README.md
|
@ -63,7 +63,106 @@ pre-commit installed at .git/hooks/pre-commit
|
||||||
|
|
||||||
3. You only need to run steps 5, 7, and 8 in the quick start during development. If the test cases need to be updated, step 6 is also needed.
|
3. You only need to run steps 5, 7, and 8 in the quick start during development. If the test cases need to be updated, step 6 is also needed.
|
||||||
|
|
||||||
## Models
|
## How does it work?
|
||||||
|
|
||||||
|
These steps are executed in runner-images. We use `sudo -u tt` to elevate the permission and run `joj3` and `joint-teapot`. All the secret files should be stored in the host machine with user `tt` and mounted into the runner (e.g. `/home/tt/.config`). Since the runner uses user `student`, we can keep the data safe.
|
||||||
|
|
||||||
|
1. Run JOJ3
|
||||||
|
1. Parse the message.
|
||||||
|
- If not specified by `-msg`, it will use the git commit message from `HEAD`. The message should meet the [Conventional Commits specification](https://www.conventionalcommits.org/). We use `scope` and `description` here.
|
||||||
|
2. Find the configuration file.
|
||||||
|
- We have `conf-root` and `conf-name` specified in the CLI argument. Then the full path of configuration file is `<conf-root>/<scope>/<conf-name>`.
|
||||||
|
3. Generate stages.
|
||||||
|
- We have an empty list of stages at the beginning.
|
||||||
|
- We check all the stages from the configuration file. Stages with empty `group` field will always be added. And stages with `group = joj` will be added when `description` contains "joj" (case insensitive).
|
||||||
|
- Every stage needs to have an unique `name`, which means if two stages have the same name, only the first one will be added.
|
||||||
|
4. Run stages.
|
||||||
|
- By default, all the stages will run sequentially.
|
||||||
|
- Each stage contains a executor and a parser. Executor (currently only sandbox) executes the command and parser parses the output generated by the executor.
|
||||||
|
- The parser can return a force quit, which means all the stages after it will be skipped.
|
||||||
|
5. Generate results.
|
||||||
|
- Once the running of stages is done, it will generate a result file where the path is specified in the configuration file.
|
||||||
|
2. Run Joint-Teapot.
|
||||||
|
1. Generally speaking, it reads the JOJ3 results file and output results on Gitea.
|
||||||
|
- We use a wrapper script `joj3-teapot` to limit the commands can be run in the original `joint-teapot` for safety.
|
||||||
|
- Currently, the environment file path and joj3 result file path are hardcoded, as `/home/tt/.config/teapot/teapot.env` and `/tmp/joj3_result.json`, respectively.
|
||||||
|
2. With `joint-teapot joj3-scoreboard`, it will update the scoreboard file in grading repo.
|
||||||
|
3. With `joint-teapot joj3-failed-table`, it will update the failed table file in grading repo.
|
||||||
|
4. With `joint-teapot joj3-create-result-issue`, it create an issue in the submitter's repo to show the results.
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
### Binaries (under `/cmd` and `/pkg`)
|
||||||
|
|
||||||
|
#### JOJ3
|
||||||
|
|
||||||
|
JOJ3 itself. Parsers and executors are compiled into the JOJ3 binary.
|
||||||
|
|
||||||
|
#### Sample
|
||||||
|
|
||||||
|
Just a sample on how to write an executable that can be called by the executor.
|
||||||
|
|
||||||
|
#### HealthCheck
|
||||||
|
|
||||||
|
The repohealth check will return a json list to for check result. The structure follows the score-comment pattern.
|
||||||
|
|
||||||
|
HealthCheck currently includes, `reposize`, `forbidden file`, `Metafile existence`, `non-ascii character` in file and message, `release tag`, and `ci files invariance` check.
|
||||||
|
|
||||||
|
The workflow is `joj3` pass cli args to healthcheck binary. See `./cmd/healthcheck/main.go` to view all flags.
|
||||||
|
|
||||||
|
### Executors (under `/internal/executors`)
|
||||||
|
|
||||||
|
#### Dummy
|
||||||
|
|
||||||
|
Do not execute any command. Just return empty `ExecutorResult` slice.
|
||||||
|
|
||||||
|
#### Sandbox
|
||||||
|
|
||||||
|
Run the commands in `go-judge` and output the `ExecutorResult` slice. Note: we communicate with `go-judge` using gRPC, which means `go-judge` can run anywhere as the gRPC connection can be established. In deployment, `go-judge` runs in the host machine of the Gitea runner.
|
||||||
|
|
||||||
|
### Parsers (under `/internal/parsers`)
|
||||||
|
|
||||||
|
#### Clang Tidy
|
||||||
|
|
||||||
|
Parser for `clang-tidy`, check `/examples/clangtidy` on how to call `clang-tidy` with proper parameters.
|
||||||
|
|
||||||
|
#### Cppcheck
|
||||||
|
|
||||||
|
Parser for `cppcheck`, check `/examples/cppcheck` on how to call `cppcheck` with proper parameters.
|
||||||
|
|
||||||
|
#### Cpplint
|
||||||
|
|
||||||
|
Parser for `cpplint`, check `/examples/cpplint` on how to call `cpplint` with proper parameters.
|
||||||
|
|
||||||
|
#### Diff
|
||||||
|
|
||||||
|
Compare the specified output of `ExecutorResult` with the content of the answer file. If they are the same, then score will be given. Just like a normal online judge system.
|
||||||
|
|
||||||
|
#### Dummy
|
||||||
|
|
||||||
|
Does not parse the output of `ExecutorResult`. It just output what is set inside the configuration file as score and comment. Currently it is used to output metadata for `joint-teapot`.
|
||||||
|
|
||||||
|
In `joint-teapot`, it will take the content before `-` of the comment of the first stage with name `metadata` as the exercise name and record in the scoreboard. (e.g. If the comment is `p2-s2-0xdeadbeef`, then the exercise name is `p2`.)
|
||||||
|
|
||||||
|
The comment in `metadata` can also be used to skip teapot commands. With `skip-teapot` in the comment, teapot will not run. And with `skip-scoreboard`, `skip-failed-table`, and `skip-result-issue`, only the corresponding step will be skipped, while the others will be executed. (e.g. If the comment is `p2-s2-0xdeadbeef-skip-scoreboard-skip-result-issue`, then only failed table step in teapot will run.)
|
||||||
|
|
||||||
|
#### Healthcheck
|
||||||
|
|
||||||
|
Parser for the `healthcheck` binary mentioned before.
|
||||||
|
|
||||||
|
#### Keyword
|
||||||
|
|
||||||
|
Match the given keyword from the specified output of `ExecutorResult`. For each match, a deduction of score is given. Can be useful if we do not have a specific parser for a code quality tool. Check `/examples/keyword`.
|
||||||
|
|
||||||
|
#### Result Status
|
||||||
|
|
||||||
|
Only check if all the status of the executor is `StatusAccepted`. Can be used to check whether the command run in the executor exit normally.
|
||||||
|
|
||||||
|
#### Sample
|
||||||
|
|
||||||
|
Parser for the `sample` binary mentioned before. Only used as a sample.
|
||||||
|
|
||||||
|
## Models (for developers only)
|
||||||
|
|
||||||
The program parses the configuration file to run multiple stages.
|
The program parses the configuration file to run multiple stages.
|
||||||
|
|
||||||
|
@ -89,78 +188,3 @@ Check the `Result` at <https://github.com/criyle/go-judge#rest-api-interface>.
|
||||||
|
|
||||||
- `Score int`: score of the stage.
|
- `Score int`: score of the stage.
|
||||||
- `Comment string`: comment on the stage.
|
- `Comment string`: comment on the stage.
|
||||||
|
|
||||||
## Binaries (under `/cmd` and `/pkg`)
|
|
||||||
|
|
||||||
### JOJ3
|
|
||||||
|
|
||||||
JOJ3 itself.
|
|
||||||
|
|
||||||
#### CLI arguments
|
|
||||||
|
|
||||||
1. `-meta-conf` to specify the meta configuration file path. JOJ3 will try to match the commit message by the `regex`, once matched, the configuration file in the `filename` will be load and run the whole process.
|
|
||||||
2. `-msg` to specify message to trigger the running. If empty, JOJ3 will use git commit message on HEAD.
|
|
||||||
|
|
||||||
### Sample
|
|
||||||
|
|
||||||
Just a sample on how to write an executable that can be called by the executor.
|
|
||||||
|
|
||||||
### HealthCheck
|
|
||||||
|
|
||||||
The repohealth check will return a json list to for check result. The structure follows the score-comment pattern.
|
|
||||||
|
|
||||||
HealthCheck currently includes, `reposize`, `forbidden file`, `Metafile existence`, `non-ascii character` in file and message, `release tag`, and `ci files invariance` check.
|
|
||||||
|
|
||||||
The workflow is `joj3` pass cli args to healthcheck binary. See `./cmd/healthcheck/main.go` to view all flags.
|
|
||||||
|
|
||||||
## Executors (under `/internal/executors`)
|
|
||||||
|
|
||||||
### Dummy
|
|
||||||
|
|
||||||
Do not execute any command. Just return empty `ExecutorResult` slice.
|
|
||||||
|
|
||||||
### Sandbox
|
|
||||||
|
|
||||||
Run the commands in `go-judge` and output the `ExecutorResult` slice.
|
|
||||||
|
|
||||||
## Parsers (under `/internal/parsers`)
|
|
||||||
|
|
||||||
### Clang Tidy
|
|
||||||
|
|
||||||
Parser for `clang-tidy`, check `/examples/clangtidy` on how to call `clang-tidy` with proper parameters.
|
|
||||||
|
|
||||||
### Cppcheck
|
|
||||||
|
|
||||||
Parser for `cppcheck`, check `/examples/cppcheck` on how to call `cppcheck` with proper parameters.
|
|
||||||
|
|
||||||
### Cpplint
|
|
||||||
|
|
||||||
Parser for `cpplint`, check `/examples/cpplint` on how to call `cpplint` with proper parameters.
|
|
||||||
|
|
||||||
### Diff
|
|
||||||
|
|
||||||
Compare the specified output of `ExecutorResult` with the content of the answer file. If they are the same, then score will be given. Just like a normal online judge system.
|
|
||||||
|
|
||||||
### Dummy
|
|
||||||
|
|
||||||
Does not parse the output of `ExecutorResult`. It just output what is set inside the configuration file as score and comment. Currently it is used to output metadata for `joint-teapot`.
|
|
||||||
|
|
||||||
In `joint-teapot`, it will take the content before `-` of the comment of the first stage with name `metadata` as the exercise name and record in the scoreboard. (e.g. If the comment is `p2-s2-0xdeadbeef`, then the exercise name is `p2`.)
|
|
||||||
|
|
||||||
The comment in `metadata` can also be used to skip teapot commands. With `skip-teapot` in the comment, teapot will not run. And with `skip-scoreboard`, `skip-failed-table`, and `skip-result-issue`, only the corresponding step will be skipped, while the others will be executed. (e.g. If the comment is `p2-s2-0xdeadbeef-skip-scoreboard-skip-result-issue`, then only failed table step in teapot will run.)
|
|
||||||
|
|
||||||
### Healthcheck
|
|
||||||
|
|
||||||
Parser for the `healthcheck` binary mentioned before.
|
|
||||||
|
|
||||||
### Keyword
|
|
||||||
|
|
||||||
Match the given keyword from the specified output of `ExecutorResult`. For each match, a deduction of score is given. Can be useful if we do not have a specific parser for a code quality tool. Check `/examples/keyword`.
|
|
||||||
|
|
||||||
### Result Status
|
|
||||||
|
|
||||||
Only check if all the status of the executor is `StatusAccepted`. Can be used to check whether the command run in the executor exit normally.
|
|
||||||
|
|
||||||
### Sample
|
|
||||||
|
|
||||||
Parser for the `sample` binary mentioned before. Only used as a sample.
|
|
||||||
|
|
Loading…
Reference in New Issue
Block a user