JOJ/JOJ3
24
5
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

Generate grades for teaching teams (scoreboard) #15

New Issue
Closed
opened 2024-03-07 09:42:11 +08:00 by 张泊明518370910136 · 23 comments
张泊明518370910136 commented 2024-03-07 09:42:11 +08:00
Owner
Copy Link
No description provided.
张泊明518370910136 added the
priority
p2
component
UI
 labels 2024-03-07 09:42:11 +08:00
张泊明518370910136 added this to the (deleted) project 2024-03-07 09:42:11 +08:00
manuel commented 2024-03-07 10:26:51 +08:00
Owner
Copy Link

simple setup:

  • server has a TT repo cloned in drone home
  • on a student release or grading keyword or any other time score is generated (eg. joj: .. or feat: .. commit) log the scores in the file
  • add/commit/push
  • on a filed stage (eg. compilation, repo health) write down the problem in a "failed" file

eg. https://focs.ji.sjtu.edu.cn/git/engr151-23fa/test/src/branch/grading

simple setup: - server has a TT repo cloned in drone home - on a student release or grading keyword or any other time score is generated (eg. `joj: ..` or `feat: ..` commit) log the scores in the file - add/commit/push - on a filed stage (eg. compilation, repo health) write down the problem in a "failed" file eg. https://focs.ji.sjtu.edu.cn/git/engr151-23fa/test/src/branch/grading
张泊明518370910136 added
priority
p0
 and removed
priority
p2
 labels 2024-04-28 11:50:26 +08:00
张泊明518370910136 changed title from Generate grades for teaching teams to Generate grades for teaching teams (scoreboard) 2024-04-28 11:50:37 +08:00
周赵嘉程521432910016 commented 2024-05-15 12:41:50 +08:00
Member
Copy Link

Not very sure about the hierarchy. Should the generator be an independent binary?

Not very sure about the hierarchy. Should the generator be an independent binary?
张佳澈520370910044 self-assigned this 2024-06-01 20:20:31 +08:00
张佳澈520370910044 commented 2024-06-01 21:36:56 +08:00
Member
Copy Link

@manuel Where is the proper place for me to put the csv scoreboard file(in the current development stage)? Maybe an empty branch in this repo, or a new empty repo?

@manuel Where is the proper place for me to put the csv scoreboard file(in the current development stage)? Maybe an empty branch in this repo, or a new empty repo?
manuel commented 2024-06-01 23:47:12 +08:00
Owner
Copy Link

@zjc_he maybe an empty branch in the test repo? eg. "grading"

@zjc_he maybe an empty branch in the test repo? eg. "grading"
张佳澈520370910044 commented 2024-06-01 23:48:02 +08:00
Member
Copy Link

👍

:+1:
张佳澈520370910044 commented 2024-06-03 12:58:36 +08:00
Member
Copy Link

@zjc_he maybe an empty branch in the test repo? eg. "grading"

@manuel What do you mean by the test repo? The test repo for 482 was archived. Or can you open a new testing repo in FOCS-dev organization?

> @zjc_he maybe an empty branch in the test repo? eg. "grading" @manuel What do you mean by the test repo? The test repo for 482 was archived. Or can you open a new testing repo in FOCS-dev organization?
manuel commented 2024-06-03 16:07:00 +08:00
Owner
Copy Link

i meant the repo you use for joj test case suite (joj-examples)

i meant the repo you use for joj test case suite (joj-examples)
张佳澈520370910044 commented 2024-06-03 16:11:46 +08:00
Member
Copy Link

That is a bit tricky... It's not so clean since it might not be how Boming designed

I'll put it there for now and add a check in main_test.go to skip that branch, @bomingzh it is good to do so?

That is a bit tricky... It's not so clean since it might not be how Boming designed I'll put it there for now and add a check in `main_test.go` to skip that branch, @bomingzh it is good to do so?
manuel commented 2024-06-03 16:12:48 +08:00
Owner
Copy Link

i can also create a joj-test repo... let me know what is best

i can also create a joj-test repo... let me know what is best
张佳澈520370910044 commented 2024-06-03 16:22:42 +08:00
Member
Copy Link

Seems boming has predicted this circumstance and that branch does not break the balance! 🎉 Everything goes fine as expected

Seems boming has predicted this circumstance and that branch does not break the balance! 🎉 Everything goes fine as expected
manuel commented 2024-06-03 16:40:04 +08:00
Owner
Copy Link

@bomingzh is too good 🎉

@bomingzh is too good :tada:
张佳澈520370910044 commented 2024-06-03 17:45:31 +08:00
Member
Copy Link

I'm currently adding up all scores in a single test, ingoring the details.

e.g, for the result:

{
      "stagename": "compile/success",
      "stageresults": [
        {
          "name": "compile",
          "results": [
            {
              "score": 0,
              "comment": ""
            }
          ]
        },
        {
          "name": "run",
          "results": [
            {
              "score": 100,
              "comment": "executor status: run time: 4488101 ns, memory: 6955008 bytes"
            },
            {
              "score": 100,
              "comment": "executor status: run time: 4973842 ns, memory: 6828032 bytes"
            }
          ]
        }
      ]
    }

I'll only write "compile/success" and total score "200" into the scoreboard(comments are trivial for scoreboard but might be used somewhere else, I'll just put here in advance)

I'm currently adding up all scores in a single test, ingoring the details. e.g, for the result: ``` { "stagename": "compile/success", "stageresults": [ { "name": "compile", "results": [ { "score": 0, "comment": "" } ] }, { "name": "run", "results": [ { "score": 100, "comment": "executor status: run time: 4488101 ns, memory: 6955008 bytes" }, { "score": 100, "comment": "executor status: run time: 4973842 ns, memory: 6828032 bytes" } ] } ] } ``` I'll only write "compile/success" and total score "200" into the scoreboard(comments are trivial for scoreboard but might be used somewhere else, I'll just put here in advance)
manuel commented 2024-06-03 18:31:43 +08:00
Owner
Copy Link

https://focs.ji.sjtu.edu.cn/git/ece482-23fa/test/src/branch/grading/p1m3-1-joj.csv

this only contains 1,0 based on whether they pass/fail tests. it seems JOJ will be able to handle more types of score? (aside of pass/fail)

is there any logic implemented to decide eg. 2.5/5 for a test case? maybe based on memory usage and/or time?

even if we don't need it now, it might good to take this type of things into account such that the design can be easily adjusted to create 1 complete grading sheet or several for correctness, cpu/mem usage, or 1 with only a summary, eg. 2.5/5 meaning correct test but slow/too much mem (details only appearing in the posted issues).

not sure what would be best for teaching team?

https://focs.ji.sjtu.edu.cn/git/ece482-23fa/test/src/branch/grading/p1m3-1-joj.csv this only contains 1,0 based on whether they pass/fail tests. it seems JOJ will be able to handle more types of score? (aside of pass/fail) is there any logic implemented to decide eg. 2.5/5 for a test case? maybe based on memory usage and/or time? even if we don't need it now, it might good to take this type of things into account such that the design can be easily adjusted to create 1 complete grading sheet or several for correctness, cpu/mem usage, or 1 with only a summary, eg. 2.5/5 meaning correct test but slow/too much mem (details only appearing in the posted issues). not sure what would be best for teaching team?
张佳澈520370910044 commented 2024-06-03 21:07:07 +08:00
Member
Copy Link

Almost finished with the first version!

Question: Where is joint-teapot deployed on the server? Can I required anyone to use JOJ3/scoreboard to have an environment variable JOINT_TEAPOT_ROOT pointing to the root dir of joint-teapot?

Almost finished with the first version! Question: Where is joint-teapot deployed on the server? Can I required anyone to use JOJ3/scoreboard to have an environment variable `JOINT_TEAPOT_ROOT` pointing to the root dir of joint-teapot?
manuel commented 2024-06-04 07:07:55 +08:00
Owner
Copy Link

nice! last fall i installed joint-pot "globally" and put an teapot.env in the course config
you just need to source the env file before running teapot (which is in the PATH, so no need to a TEAPOT_ROOT variable i guess?)

nice! last fall i installed joint-pot "globally" and put an `teapot.env` in the course config you just need to source the `env` file before running teapot (which is in the `PATH`, so no need to a `TEAPOT_ROOT` variable i guess?)
张佳澈520370910044 commented 2024-06-05 13:01:20 +08:00
Member
Copy Link

@manuel Could you please clone the JOJ3-example repo on the drone server and put it in parallel with the JOJ3 repo? Specifically here in my local client:

├── JOJ3
├── JOJ3-examples

Or for another option, you may want to clone the JOJ3-example repo every time you want to upload the scoreboard, and then remove it?

@manuel Could you please clone the JOJ3-example repo on the drone server and put it in parallel with the JOJ3 repo? Specifically here in my local client: ``` ├── JOJ3 ├── JOJ3-examples ``` Or for another option, you may want to clone the JOJ3-example repo every time you want to upload the scoreboard, and then remove it?
manuel commented 2024-06-05 13:10:00 +08:00
Owner
Copy Link

just the grading branch?

just the grading branch?
张佳澈520370910044 commented 2024-06-05 13:12:07 +08:00
Member
Copy Link

Doesn't matter, teapot will detect and switch to the grading branch

Also for the second option cloneing and removing the repo is also done automatically by teapot

Doesn't matter, teapot will detect and switch to the grading branch Also for the second option cloneing and removing the repo is also done automatically by teapot
张佳澈520370910044 commented 2024-06-05 17:27:54 +08:00
Member
Copy Link

No need anymore, I implemented option 2

No need anymore, I implemented option 2
张佳澈520370910044 commented 2024-06-09 09:11:02 +08:00
Member
Copy Link

@manuel We are done on teapot side, can you pull the latest version of teapot so we can push forward to do testing?

@manuel We are done on teapot side, can you pull the latest version of teapot so we can push forward to do testing?
张佳澈520370910044 commented 2024-06-09 21:39:45 +08:00
Member
Copy Link

It seems that there are problems finding environmental variable ORIG_HOME when running drone

error message from drone: https://focs.ji.sjtu.edu.cn:2222/FOCS-dev/JOJ3/322/1/4 line 328
code: https://focs.ji.sjtu.edu.cn/git/FOCS-dev/JOJ3/src/branch/scoreboard/internal/scoreboard/scoreboard.go#L57

It seems that there are problems finding environmental variable `ORIG_HOME` when running drone error message from drone: https://focs.ji.sjtu.edu.cn:2222/FOCS-dev/JOJ3/322/1/4 line 328 code: https://focs.ji.sjtu.edu.cn/git/FOCS-dev/JOJ3/src/branch/scoreboard/internal/scoreboard/scoreboard.go#L57
manuel commented 2024-06-10 22:15:19 +08:00
Owner
Copy Link

soryr had not seen that... have you pulled the newest teapot?

soryr had not seen that... have you pulled the newest teapot?
张佳澈520370910044 commented 2024-06-10 23:17:11 +08:00
Member
Copy Link

Boming fixed for me, now it can run good on drone

Boming fixed for me, now it can run good on drone
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
bug

Something is not working

  
component
executor
executors in different stages

  
component
framework
software architecture

  
component
parser
parsers in different stages

  
component
UI
drone, gitea, and config interfaces

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
priority
p0

  
priority
p1

  
priority
p2

  
priority
p3

  
question

More information is needed

  
wontfix

This won't be fixed

No Label
bug
component
executor
component
framework
component
parser
component
UI
duplicate
enhancement
help wanted
invalid
priority
p0
priority
p1
priority
p2
priority
p3
question
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
张佳澈520370910044
4 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: JOJ/JOJ3#15
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?