Update [Admin] JOJ server setup

%5BAdmin%5D-Course-container-setup-%5BWIP%5D.md

@@ -1,15 +0,0 @@
-# Course container setup
-
-## Container deployment
-
-1. Create a new `bot-${REPO_OWNER_LOWER}` user on Gitea
-2. Generate a new token for `bot-${REPO_OWNER_LOWER}` with permissions:
-    - issue `rw`
-    - repository `rw`
-    - organizaion `r`
-3. On `${REPO_OWNER_LOWER}-joj` repo settings, add the gitea token of `bot-${REPO_OWNER_LOWER}` as value to `Settings->Actions->Secrets->Add Secret`, with name `TEAPOT_GITEA_TOKEN`
-4.  Edit `root/deploy.conf` based on needed software/tools
-5.  Deploy a new container for the course
-    - create a zfs volume
-    - setup permissions
-    - deploy

%5BAdmin%5D-JOJ-server-setup.md

@@ -0,0 +1,87 @@
+# JOJ server setup guide
+
+## Host setup
+
+Prepare unprivileged lxc containers
+
+- `apt install sudo zfs-dkms zfsutils-linux jq lxc uidmap`
+- ssh setup:
+    - generate a new ssh key `ssh-keygen -t ed25519`
+    - add to bot-joj
+- clone joj3-hs `git clone ssh://git@focs.ji.sjtu.edu.cn:2222/JOJ/JOJ3-hs.git`
+- copy joj3-hs/admin/config
+    - `/etc/sudoers.d/joj-deploy`
+    - `/etc/subuid`
+    - `/etc/subgid`
+    - `/etc/lxc/lxc-usernet`
+    - `/home/ja/.config/lxc/default.conf`
+- as `ja` user 
+    - `mkdir -p ~/.local/share/lxc`
+    - `chmod o+x ~/.local/share/lxc`
+
+
+## Guest setup
+
+Before setting up a JOJ container for COURSE, ensure the following exist and are properly setup:
+
+- bot-COURSE user
+- COURSE organisation
+- COURSE-joj repo
+
+All actions are performed as `ja`.
+
+- Clone the course-joj repo
+- Edit etc/joj-container-config.conf based on needed software/tools
+- Run `joj-container-deploy COURSE` 
+
+*Note.* the ssh key, act_runner token, and teapot token are backed up in `~/courses/COURSE`
+
+### Advanced go-judge setup
+
+Network access: 
+- add `-net-share` to service file 
+- add `/etc/ssl` to `/etc/go-judge/mount.yaml`
+
+*Note.* used in sfocs-joj
+
+Mount directories:
+- edit `/etc/go-judge/mount.yaml`
+- permissions might need to adjusted, eg. group `nogroup` with write access if mounting a directory from `$HOME`
+
+*Note.* used in `sfocs-joj` (elm-packages) and `ece477-joj` (latex font setup)
+
+
+## Trouble shooting
+
+### Containers stop automatically
+
+Fixed using `loginctl enable-linger frown`
+
+### Mounting `sysfs` fails
+
+- check umask from `/proc/xxx/status` where `xxx` is the `pid` of systemd --user
+- adjust `/etc/pam.d/common-session-noninteractive` with content
+```
+session optional                        pam_umask.so umask=0002
+```
+
+### lxc-copy fails
+
+Reason: `apparmor` bug in some debian versions
+
+Edit `/etc/apparmor.d/usr.bin.lxc-copy`
+
+```
+  mount options=(rw,move) -> /home/ja/.local/share/lxc/{,**},
+```
+
+### lxc fails to assign CPUs
+
+On the host run:
+
+```
+echo "+cpuset" > /sys/fs/cgroup/user.slice/cgroup.subtree_control
+echo "+cpuset" > /sys/fs/cgroup/user.slice/user-1000.slice/cgroup.subtree_control
+echo "+cpuset" > /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/cgroup.subtree_control` 
+echo "+cpuset" >  /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/cgroup.subtree_control
+```