Shell to Python Continuous Deployment

Get a common set of Python based CI / CD commands
from various contexts of CA, DNS, CI and OS / OCI.

---

Why

Project workflows can occur in:

• an online or offline context, dealing with custom CA & DNS servers
• various Continuous Integration platforms, with different workflow syntaxes
• specific Operating Systems, having different commands and package names
• specific containers for JavaScript based actions (ForgeJo, Gitea, GitHub)

What if there was a unified way to do so?

---

How

Picture it…

Group environment variables

• numbered
  • Certificate Authority armored certificates
  • Domain Name System servers IP addresses
• ref names for deployment contexts of feature, staging & release
• alternate locations of
  • Operating Systems & Software packages
  • Git repositories of
    • this project
    • its parent framework
    • projects not available in some distributions
      • shunit2
• SSH key & hosts for deployments
• a standalone POSIX shell script payload

Workflow files

• calling the POSIX shell script payload as their very first step
• using a unified YAML syntax made of actions implemented in Python

Integration & Deployment

Shell

• setting up the chosen Operating System container
  • Domain Name System resolving
  • package manager configuration & repositories
  • Certificate Authorities trusting
  • Operating System updating
• installing Git to clone
  • this project
  • its parent framework
• installing both Python
  • system environment
  • virtual environment
• generating a Python module to switch context

Python

• setting up an OpenSSH client
  • known hosts
  • private key
• and finally parsing commands from
  • system
  • workflow
• to run available actions
  • check project
  • build project
  • browse workspace
  • synchronize

---

What

Features

Handle project workflows in a unified way:

• whether the network infrastructure is

  • online
  • offline with custom
    • Certificate Authorities
    • Domain Name System servers

• whatever the Continuous Integration platform

  • ForgeJo
  • Gitea
  • GitHub
  • GitLab
  • SourceHut

• whatever the Operating System container

  • Alma
    • 9
    • 8
  • Alpine
    • 3.20
    • 3.19
  • Arch
    • 20240818 (.0.255804)
    • 20240101 (.0.204074)
  • Debian
    • Bookworm (12)
    • Bullseye (11)
  • Fedora
    • 40
    • 39
  • OpenSUSE
    • 15.6
    • 15.5
  • Rocky
    • 9
    • 8
  • Ubuntu
    • Noble (24.04)
    • Jammy (22.04)

Environment variables

Variable	Description	Default
SPCD_CA_n	Numbered CA certificates
SPCD_DNS_n	Numbered name servers	9.9.9.9
SPCD_GIT_RWX	RWX Git repository	rwx
SPCD_GIT_SHUNIT	ShUnit Git repository	shunit2
SPCD_GIT_SPCD	SPCD Git repository	spcd
SPCD_REF_FEATURE	Feature deployment ref	f
SPCD_REF_RELEASE	Release deployment ref	main
SPCD_REF_STAGING	Staging deployment ref	dev
SPCD_SSH_HOSTS	domain.tld ssh-type pub
SPCD_SSH_KEY	SSH private key
SPCD_TXT_LOCALE	Locale to use for text	en_US
SPCD_URL_ALMA	Alma repository URL	https://repo.almalinux.org
SPCD_URL_ALPINE	Alpine repository URL	https://dl-cdn.alpinelinux.org
SPCD_URL_ARCH	Arch repository URL	https://geo.mirror.pkgbuild.com
SPCD_URL_DEBIAN	Debian repository URL	https://deb.debian.org
SPCD_URL_EPEL	EPEL repository URL	https://dl.fedoraproject.org
SPCD_URL_FEDORA	Fedora repository URL	https://rpmfind.net
SPCD_URL_OPENSUSE	OpenSUSE repository URL	https://download.opensuse.org
SPCD_URL_PYTHON	Python repository URL	https://pypi.org
SPCD_URL_ROCKY	Rocky repository URL	https://dl.rockylinux.org
SPCD_URL_UBUNTU	Ubuntu repository URL	https://ubuntu.mirrors.ovh.net

Operating Systems containers

Latest

os	https	up ca	python	graphviz	plantuml	shellcheck	shunit	shfmt	gource	ffmpeg
Arch 20240818	☑	☑	3.12	12.0	1.2023.13	0.10	2.1.8	3.8	0.54	7.0.2
Alpine 3.20	☑	☐	3.12	9.0	1.2024.4	0.10	2.1.8	3.8	0.54	6.1.1
Fedora 40	☑	☑	3.12 → 3.13	9.0	1.2024.6	0.9	2.1.6	3.7	0.55	6.1.2
Debian Bookworm	☐	☐	3.11	2.42	1.2020.2	0.9	2.1.8	3.6	0.54	5.1.6
OpenSUSE 15.6	☐	☑	3.6 → 3.12	2.48	1.2020.9	0.8	2.1.6	3.5	0.54	4.4.4
Ubuntu Noble	☐	☐	3.12	u2.42	u1.2020.2	u0.9	2.1.8	u3.8	0.54	6.1.1
Alma / Rocky 9	☑	☑	3.9 → 3.12	2.44	e1.2024.6	e0.8				e5.1.4

Previous

os	https	up ca	python	graphviz	plantuml	shellcheck	shunit	shfmt	gource	ffmpeg
Alpine 3.19	☑	☐	3.11	9.0	1.2023.12	0.9	2.1.8	3.7	0.54	6.1.1
Fedora 39	☑	☑	3.12 → 3.13	8.1	1.2024.6	0.9	2.1.6	3.5	0.55	6.1.1
OpenSUSE 15.5	☐	☑	3.6 → 3.11	2.48	1.2020.9	0.8	2.1.6	3.5	0.54	4.4.4
Alma / Rocky 8	☑	☑	3.6 → 3.12	2.40	e1.2024.6	e0.6

Older Python

os	https	up ca	python	graphviz	plantuml	shellcheck	shunit	shfmt	gource	ffmpeg
Ubuntu Jammy	☐	☐	3.10	u2.42	u1.2020.2	u0.8	2.1.6	u3.4	0.51	4.4.2
Debian Bullseye	☐	☐	3.9	2.42	1.2020.2	0.7	2.1.6		0.51	4.3.7

---

Who

By

• Marc Beninca

For

• People feeling the need to aim for consistency in the CI / CD universe

---

Where

Chat

• Discord
• IRC

Forge

• Repository
• RSS
• Workflows

Deployment

• Site

---

When

Tasks for a 1st stable release

Documentation

• build instead of readme
• generate graphs
• list behavior differences
  • Continuous Integration platforms
  • Operating Systems

Shell

• comment
• handle errors
• packages
  • configure
    • apk
    • pacman
    • zypper
  • repositories
    • codecs
      • epel
      • fedora
      • opensuse
    • opensuse
      • disable & enable https
• locales
  • persist
• test
• workspace variable

Shell → Python

• check file & variable sums
• install system packages
  • epel
    • plantuml
    • shellcheck
  • graphviz
  • openssh
  • rsync
  • shfmt
  • shunit
• python virtual environment
  • install system package
  • configure pip
  • venv
    • create
    • activate
• query package
• relay environment module name

Python

• check
  • .py
  • .sh
    • maximum line length
• comment
• detect ssh private key type
• fit banner to text
• implement substeps
• list sys.path
• set workflow templates
  • actions
  • includes
• try git repo url variable first for shunit
• test

Task stack

• automate versions fetching
• gource, xvfb, xauth
• handle openh264 repositories
• tex
• translate to french
• try to support nix