Video Specials

Set-Up on Debian

General

As time of writing, there is no Debian repository or package or other OS for that matter, so you have to do it all yourself. YADE is a package for managed file transfer. It can be used stand alone or within JobScheduler, however JS does not include managed file transfer without YADE. So it is sort of a plugin that can be used stand alone too. I will not document that explicitely unless it deviates from the one of JobScheduler, but from the obvious.

This page describes bare metal installation, no containerisation of any kind (docker, kubernetes, you name it). Maybe some day, I pack this into Ansible. I recommend to issue command after command instead of as script as it is easier to amend a failure immediately than doing it in the aftermath for all dependent steps. This cookbook creates three instances A (acceptance test), D (development) and I (integration test).

Architecture

I want to setup instances as much as a module as possible to be able to move/copy it to other machines or within machines with as little effort as possible. So I decided to use a binary directory base /opt/jobscheduler/bin with version subdirectories and to provide symlinks to the specific version in the instance directories /opt/jobscheduler/<instance_name>. In such way, I can have different versions on the same computer for different environments, making it easier to upgrade.

I will setup template structures that I will instanciate for accpetance, integration and development.

What ever I put forward, amend to your needs.

tree /opt/jobscheduler -L 4
/opt/jobscheduler
├── A
│   ├── agent -> ../bin/7/2.7.3/agent
│   ├── agent.service
│   ├── agent_instance.sh
│   ├── controller -> ../bin/7/2.7.3/controller
│   ├── controller.service
│   ├── controller_instance.sh
│   ├── joc -> ../bin/7/2.7.3/joc
│   ├── joc.service
│   ├── joc_instance.sh
│   ├── tablespaces
│   │   └── 7                                                                                                                   
│   │       └── 2.7.3                                                                                                           
│   └── yade -> ../../yade/bin/1.13.23
├── D
│   ├── agent -> ../bin/7/2.7.3/agent
│   ├── agent.service
│   ├── agent_instance.sh
│   ├── controller -> ../bin/7/2.7.3/controller
│   ├── controller.service
│   ├── controller_instance.sh
│   ├── joc -> ../bin/7/2.7.3/joc
│   ├── joc.service
│   ├── joc_instance.sh
│   ├── tablespaces
│   │   └── 7                                                                                                                   
│   │       └── 2.7.3                                                                                                           
│   └── yade -> ../../yade/bin/1.13.23
├── I
│   ├── agent -> ../bin/7/2.7.3/agent
│   ├── agent.service
│   ├── agent_instance.sh
│   ├── controller -> ../bin/7/2.7.3/controller
│   ├── controller.service
│   ├── controller_instance.sh
│   ├── joc -> ../bin/7/2.7.3/joc
│   ├── joc.service
│   ├── joc_instance.sh
│   ├── tablespaces
│   │   └── 7                                                                                                                   
│   │       └── 2.7.3                                                                                                           
│   └── yade -> ../../yade/bin/1.13.23
├── bin                                                                                                                         
│   └── 7                                                                                                                       
│       └── 2.7.3                                                                                                               
│           ├── agent                                                                                                           
│           ├── controller                                                                                                      
│           └── joc
└── template
    ├── agent -> ../bin/7/2.7.3/agent
    ├── agent.service
    ├── agent_instance.sh
    ├── controller -> ../bin/7/2.7.3/controller
    ├── controller.service
    ├── controller_instance.sh
    ├── joc -> ../bin/7/2.7.3/joc
    ├── joc.service
    ├── joc_instance.sh
    ├── tablespaces
    │   └── 7                                                                                                                   
    │       └── 2.7.3                                                                                                           
    └── yade -> ../../yade/bin/1.13.23

The repository of JOC shall be stored in a seperate database for each instance. The tablespace directories take care partly of this decision.

Links

Templates and common structures

Some of the structures get used in the instances as they are created in this part, e.g. the database user.

Repository

I will be using PostgreSQL, a very sophisticated OpenSource database, to have JobScheduler store its repository into. I presume, the base installation has been done already. The file system folders for its tablespaces need to be created now.

We first create a template database. The work databases are cloned from it. That way, creating a database is quickly done. However, it might actually only be useful if you have/want to have several environments on one metal. In any case, the overhead is minimal, and if you need it, you will be glad to have it.

Create the template database

Connect with superuser power like postgres

sudo -u postgres psql -p 5437

Create owner and template database

create role JS_JOC
   login
   password 'replace_me';
create database JS␟7␟2_7_3␟TEMPLATE
   owner = JS_JOC
   encoding = 'UTF8'
   lc_collate = 'en_GB.UTF-8' -- must be compatible with the template DB
   lc_ctype = 'en_GB.UTF-8' -- must be compatible with the template DB
   template = TEMPLATE1;

Finalise the template database

Connect with your JobScheduler user to your new template database (you might want to place the password for it in ~/.pgpass). The case of the database name (-d option) is important. Unless you created your database with its name within double quotes, you need to use lower case. Otherwise, you need to repeat the case exactly.

psql -p 5437 -U js_joc -d js␟7␟2_7_3␟template

Create a schema, make it the user's default and install

create schema JS_JOC authorization JS_JOC;
alter role JS_JOC set search_path to JS_JOC, public;
alter database JS␟7␟2_7_3␟TEMPLATE is_template TRUE;

Set-up OS

OS User

The services will run under different system user for the respective environment, segregation of power that is. If a service gets moved to a machine of its own, one can change the user name if that is of interest.

Commands to it

I am fond of the z-shell, and a home directory is not needed as we do not want the user to get sudoed to anyway. Sure, there are work arounds to that and maybe, if you want to have a .zshrc, you need a home, but for the time being, no. Amend to your specifics.

sudo adduser --system --no-create-home --shell /usr/bin/zsh --group js-agent_A
sudo adduser --system --no-create-home --shell /usr/bin/zsh --group js-joc_A
sudo adduser --system --no-create-home --shell /usr/bin/zsh --group js-controller_A
sudo adduser --system --no-create-home --shell /usr/bin/zsh --group js-agent_D
sudo adduser --system --no-create-home --shell /usr/bin/zsh --group js-joc_D
sudo adduser --system --no-create-home --shell /usr/bin/zsh --group js-controller_D
sudo adduser --system --no-create-home --shell /usr/bin/zsh --group js-agent_I
sudo adduser --system --no-create-home --shell /usr/bin/zsh --group js-joc_I
sudo adduser --system --no-create-home --shell /usr/bin/zsh --group js-controller_I

File System

Commands to it, part 1

sudo mkdir -p /opt/jobscheduler/template/tablespaces/7/2.7.3
sudo chown -R postgres:postgres /opt/jobscheduler/template/tablespaces
sudo mkdir -p /opt/jobscheduler/bin/7/2.7.3
# for the agent and controller binaries unpack the downloaded binaries into this new directory and amend the name of the top directories as you see fit
# for the JOC unpack it to a directory of your liking but not within this tree. After the run of the setup.sh it can be removed.

Properties file

It will be used only once. You might want to craete it within the JOC unpack directory.
# Jetty configuration
jetty.home=/opt/jobscheduler/bin/7/2.7.3/joc
jetty.base=/opt/jobscheduler/template/jetty_base
jetty.logging.level=INFO                         # Logging-Level
jetty.logs=/opt/jobscheduler/template/logs/jetty # Logverzeichnis

# JOC configuration
install.user=js-joc_TEMPLATE   # OS user
joc.title=JobScheduler Template (T)
joc.instance.id=0              # 0, template, 100 - 199 development, 200 - 299 integration tests, 300 - 399 acceptance test, 400 - 499 production
api.server=true                # must be true if you want to have the scheduling functionality
memory.xmx=512
memory.xms=256
hibernate=true                 # not sure what happens, if false
security.level=MIN             # for production, maybe acceptance to MAX might be better

# Database configuration
# Be aware that the DB and all have to be setup beforehand.
db.driver=org.postgresql.Driver
db.url=jdbc:postgresql://localhost:5437/js␟7␟2_7_3␟template  # DB-URL
db.username=js_joc
db.password=REPLACE_ME

Commands to it, part 2


cd /opt/jobscheduler/A
sudo ln -sf ../bin/7/2.7.3/agent agent
sudo cp -a ../bin/7/2.7.3/agent/bin/agent.service-example agent.service
sudo cp -a ../bin/7/2.7.3/agent/bin/agent_instance.sh-example agent_instance.sh
sudo ln -sf ../bin/7/2.7.3/controller controller
sudo cp -a ../bin/7/2.7.3/controller/bin/controller.service-example controller.service
sudo cp -a ../bin/7/2.7.3/agent/bin/controller_instance.sh-example controller_instance.sh
sudo ln -sf ../bin/7/2.7.3/joc joc
sudo cp -a ../bin/7/2.7.3/joc/bin/joc.service-example joc.service # 2.7.3 does NOT provide an example file. In such case, create one (see below) before you continue
sudo cp -a ../bin/7/2.7.3/agent/bin/joc_instance.sh-example joc_instance.sh # 2.7.3 does NOT provide an example file. In such case, create one (see below) before you continue
sudo rsync -a --ignore-existing --exclude='tablespaces/*' /opt/jobscheduler/A /opt/jobscheduler/D # development environment
sudo rsync -a --ignore-existing --exclude='tablespaces/*' /opt/jobscheduler/A /opt/jobscheduler/I # integration test environment
sudo ln -sf /opt/jobscheduler/A/agent.service /etc/systemd/system/agent_A.service
sudo ln -sf /opt/jobscheduler/A/controller.service /etc/systemd/system/controller_A.service
sudo ln -sf /opt/jobscheduler/A/joc/bin/joc.service /etc/systemd/system/joc_A.service
sudo ln -sf /opt/jobscheduler/D/agent.service /etc/systemd/system/agent_A.service
sudo ln -sf /opt/jobscheduler/D/controller.service /etc/systemd/system/controller_A.service
sudo ln -sf /opt/jobscheduler/D/joc.service /etc/systemd/system/joc_A.service
sudo ln -sf /opt/jobscheduler/I/agent.service /etc/systemd/system/agent_A.service
sudo ln -sf /opt/jobscheduler/I/controller.service /etc/systemd/system/controller_A.service
sudo ln -sf /opt/jobscheduler/I/joc.service /etc/systemd/system/joc_A.service
# no production environment here as I have this on another computer

You will have noticed the tablespaces directory. It is for the database. So every environment is in one file tree. It stands to reason that moving an environment to another computer is going to be easier that way, at least cold backup should be.

Kommentare

Kommentar veröffentlichen