Skip to content

FEATURE.md

Latest commit

 

History

History
377 lines (264 loc) · 21.6 KB

FEATURE.md

File metadata and controls

377 lines (264 loc) · 21.6 KB
Raw

Feature Branch Deployment

The feature branch deployment describes the deployment and initialization process of multiple application instances on the same host.

If you want to use the feature branch deployment functionalities add the belonging autoloader to your project deploy.php:

require_once(__DIR__ . '/vendor/move-elevator/deployer-tools/deployer/feature/autoload.php');

See an example configuration here: feature example.

Important: if you are using the deployer recipe for a specific application (e.g. symfony), the autoload for the feature branch deployments needs to be placed after the application autoload.

The following steps are necessary to successfully setup the deployment workflow:

Prerequirements

Choose an according database management type for your application.

Configure your web server (Apache or nginx) to serve feature branch instances.

Add the following line to your deployer host entry, to enable the feature branch deployment for this stage:

labels:
   type: feature-branch-deployment

For using the synchronization, you need a sync tool installed. The preferred, Composer-native option is php-sync-tool:

$ composer require --dev konradmichalik/php-sync-tool

It is auto-detected at vendor/bin/sync-tool and used automatically once installed, no deploy.php change needed. The legacy Python tools remain supported as a fallback for projects that have not migrated yet:

$ pip3 install db-sync-tool-kmi file-sync-tool-kmi

Initialization

The feature:setup command represent the initialization of a new feature branch. It creates the necessary folder structure and database for the application. Also it extends the given host information with the necessary dynamic feature instance information.

Important: Note, that you initially need to import the database of the reference state manually (which all other feature branches sync the database from).

$ vendor/bin/dep feature:setup stage --feature=TEST-01

The --feature= value may be a full branch name. Path separators are replaced by a hyphen so the instance stays a single flat directory, url segment and database suffix:

--feature= instance name instance name, subdomain mode
TEST-01 TEST-01 test-01
feature/TEST-01 feature-TEST-01 feature-test-01
bugfix/TEST-01 bugfix-TEST-01 bugfix-test-01
release/1.2.0 release-1.2.0 release-1-2-0

Names that already consist of letters, digits, _, - and . are used unchanged, so existing instances and their databases stay reachable. Since the branch prefix is kept, feature/TEST-01 and bugfix/TEST-01 remain two separate instances. The same normalization is applied by feature:cleanup when it compares remote git branches with the deployed instances.

With feature_url_pattern set (see Subdomain mode), the instance name additionally becomes a valid DNS hostname label: lowercased, . and _ replaced by -, and truncated with a short hash suffix if it would otherwise exceed 63 characters.

A --feature= value that yields no usable name is rejected instead of falling back to the base instance, which the feature scaffolding would otherwise overwrite. That covers values without any allowed character as well as . and ...

Because the separator is replaced rather than encoded, feature/TEST-01 and a branch literally named feature-TEST-01 map to the same instance. Do not use both spellings for different branches in one project.

The recipe already wires this task into the deploy flow, together with a feature:init before deploy:info. That ordering matters: deploy:info resolves {{release_name}} and deployer caches the result for the rest of the run, so the feature instance has to be known before it runs. Do not hook feature:setup any earlier yourself.

Upgrading: if your deploy.php carries a before('deploy:info', 'feature:init') (or an equivalent feature:setup hook) as a workaround for that ordering, remove it — the recipe registers it now and the hook would otherwise run twice.

If the application needs to setup additional configuration files for e.g. storing the database credentials, use the feature templates to provide this kind of dynamic setup. For example the TYPO3 setup with a .env file:

set('feature_templates', [
    __DIR__ . '/.deployment/deployer/templates/.env.dist' => '/shared/.env'
]);

This configuration defines the local template file as well as the remote target destination. Within the template file a bunch of predefined variables are available and can be placed with {{attribute}}.

attribute description
DEPLOYER_CONFIG_DATABASE_HOST default is 127.0.0.1, overwrite with deployer set('database_host', '127.0.0.1');
DEPLOYER_CONFIG_DATABASE_PORT default is 3306, overwrite with deployer set('database_port', '3306');
DEPLOYER_CONFIG_DATABASE_USER should be defined with database_user in the host configuration
DEPLOYER_CONFIG_DATABASE_NAME will be dynamically generated
DEPLOYER_CONFIG_FEATURE_NAME the normalized instance name derived from the --feature= command line argument
DEPLOYER_CONFIG_FEATURE_URL will be dynamically generated, e.g. https://test.local/TEST-01/ or, in subdomain mode, https://test-01.stage.example.com/
DEPLOYER_CONFIG_FEATURE_PATH will be dynamically generated, e.g. TEST-01/current/public/; blank in subdomain mode, since the app is served from its subdomain's root

You can extend these list be providing more environment variables starting with DEPLOYER_CONFIG_*.

Hint: If you're using other deployer commands within the feature branch deployment context, you should use the feature:init task to extend the host definition with the necessary feature instance configuration:

before('my:task', 'feature:init');

feature:init is a no-op without --feature, so the command keeps operating on the base instance. Use feature:select instead if the command should offer an interactive choice between the existing feature instances when --feature is omitted:

before('my:task', 'feature:select');

rollback and the debug:* tasks are already wired to feature:select by the recipe.

Deletion

The feature:stop command deletes the feature branch instance, including the folder structure and the database.

$ vendor/bin/dep feature:stop stage --feature=TEST-01

Hooks

The recipe provides two no-op extension points for external infrastructure a feature instance needs beyond its deploy directory and database, most commonly registering or removing a subdomain via a hosting provider's API when subdomain mode is used. Override them in your project's deploy.php:

task('feature:provision', function () {
    $host = parse_url(get('public_urls')[0], PHP_URL_HOST);
    // e.g. call your provider's API to create $host and wait until it resolves
});

task('feature:deprovision', function () {
    // e.g. remove it again
});

feature:provision runs once per new instance, right after its database is created (feature:setup) and before feature_templates are rendered, since a template may already depend on the infrastructure it provisions. feature:deprovision runs before an instance's symlink, database and directory are removed, for both feature:stop and feature:cleanup.

Both must be idempotent: an aborted deploy can leave feature:provision applied without the instance directory existing yet, so a retried deploy runs it again; deletion can likewise be invoked more than once for the same instance.

Notification

The feature:notify command notifies about a successful initial feature branch deployment and can be configured with the following settings:

set('feature_msteams_color', '#0097A7');
set('feature_msteams_text', 'Der Branch **[{{feature}}]({{public_url}})** wurde bereitgestellt.');

This task should be placed right before finishing the deployment task.

before('deploy:success', 'feature:notify');

Hint: it is possible to mute the notification by providing the environment variable DEPLOYER_CONFIG_NOTIFICATION_MUTE.

Synchronization

The feature:sync command synchronizes the feature branch instance with data from a different instance using the db-sync-tool.

You need to configure the paths within your deploy.php to the db-sync-tool configuration files:

# general sync configuration file for feature branches
set('feature_sync_config', __DIR__ . '/.deployment/db-sync-tool/sync-stage-to-feature.yaml');
# where to find the sync target file
set('feature_sync_target_path', '{{deploy_path}}/shared/.env.local');
# where to find the sync file directory
set('feature_sync_target_path_files', '{{deploy_path}}/shared/public/upload/media/');

Having an optionally entry files in your sync configuration (e.g. sync-stage-to-feature.yaml) enables the file-sync-tool for a simple file synchronisation feature.

It's possible to override db_sync or file_sync command (setting it to false disables the feature):

# disables db_sync feature entirely
set('db_sync_tool', false);
# disables file_sync feature entirely
set('file_sync_tool', false);

Periodic sync

This function can be used as periodic sync between instances (e.g. nightly sync between the production system and a specific feature branch instance on the stage system).

Therefore you should use the feature:sync:standalone task, which includes the feature:sync command and may also include additional (application specific) commands like clearing the cache.

For overwriting the default feature_sync_config setting, add an additional environment variable:

DEPLOYER_CONFIG_FEATURE_SYNC_CONFIG=./.deployment/db-sync-tool/sync-prod-to-feature.yaml
$ vendor/bin/dep feature:sync:full stage --feature=test

Information

The feature:list command lists all available feature branches:

$ vendor/bin/dep feature:list

+----------------+------------------ stage ------------------------------------+
| Feature Branch | Modification Date | Public URL                              |
+----------------+-------------------+-----------------------------------------+
| feature-start  | 20.01.2023 15:07  | https://test.example.local/feature-start   |
| test           | 20.01.2023 17:45  | https://test.example.local/test            |
+----------------+-------------------+-----------------------------------------+

Also a simple feature branch overview index application can be placed on the hosts with the feature:index command:

$ vendor/bin/dep feature:index stage

This uploads the index application to the host and creates a symbolic link to the web root. The folder structure on the server will look like this:

├── .fbd/
│   ├── index/
│   ├── index.php
│   ├── index.config.php
│   ├── .htaccess
│   ├── logo.svg or logo.png (optionally, SVG preferred)
│   └── background.png (optionally)
└── index.php -> .fbd/index.php

The index configuration, including feature_index_jira_auth, is stored as index.config.php so the web server executes it instead of serving it as plain text. Cached Jira responses below index/var/ are PHP files for the same reason. Earlier versions stored a plain index.json and raw JSON cache files there, feature:index removes both.

To hide the credentials from other local users, index.config.php is assigned to requirements_user_group (the web server group, www-data by default) and set to mode 640. If the deploy user cannot change the group, the file stays at 644 and feature:index warns when feature_index_jira_auth is set.

On Apache, the shipped .htaccess files additionally deny HTTP access to index.config.php, index.json, index/src/ and index/var/. On nginx, add the equivalent rule to the server block, see Web server.

Issue details (type, status, assignee) are only requested from Jira when feature_index_jira_api is set (e.g. https://acme.atlassian.net/rest/api/3/issue/, empty by default). Non-public Jira instances additionally need feature_index_jira_auth (base64 encoded email:api-token). Failed requests are neither shown nor cached, the remaining issues are requested in parallel.

Per instance the index shows:

  • the Jira issue summary, status, type and assignee
  • the last deployment (relative, absolute time as tooltip), the deploying user and the release, read from Deployer's .dep/releases_log. The user is omitted for names listed in feature_index_hide_deploy_users (matched case-insensitively, defaults to ['ci']), e.g. a CI service account that deploys every release and so carries no information
  • a locked badge while .dep/deploy.lock exists, i.e. a deployment is running or was aborted
  • a stale badge for feature instances without a deployment for feature_index_stale_days days (default 14, 0 disables it) or with a done Jira issue, as a hint for feature:stop or feature:cleanup
  • links to the backend (feature_index_backend_path, typo3/ for TYPO3), the git branch (feature_index_git_branch) and a button copying the instance URL

Instances are grouped into the stage (the instances listed in feature_stop_disallowed_names, main and master by default), releases (release-x.y.z) and feature instances (everything else, with or without an issue key, most recently deployed first). A summary line counts instances, stale and locked ones and shows the disk usage, a filter narrows the list by branch, issue key or summary. The page follows the system light or dark mode.

The index page requires PHP 8.1 on the host.

Pathing

Normally the feature instances are deployed in the web root of the host, e.g. /var/www/html/. But the entry point of an application could lie in a nested folder structure, e.g. /var/www/html/app/current/public/, so the url would look like this: https://test.local/app/current/public/.

To make this a bit easier, you can use the feature:urlshortener command. Initially activate the function via:

set('feature_url_shortener', true);
after('deploy:symlink', 'feature:urlshortener');

The folder structure on the server will look like this:

├── .fbd
│   └── instances
│       ├── app
│       │   └── current
│       │       └── public/
│       └── ...
└── app -> .fbd/instances/app/current/public

So the resulting url will look like: https://test.local/app.

Subdomain mode

Instead of a subpath, a feature instance can be served from its own subdomain. Set feature_url_pattern with a <feature> placeholder:

set('feature_url_pattern', 'https://<feature>.stage.example.com/');

This is unset ('') by default, which keeps every existing project on the path-based behaviour above, byte-identical. Once set, public_urls, DEPLOYER_CONFIG_FEATURE_URL, feature:list and the feature index page all resolve to the subdomain instead. DEPLOYER_CONFIG_FEATURE_PATH becomes blank and FEATURE_BRANCH_PATH_PUBLIC (see build recipe) is not set, since the application is served from the root of its own subdomain rather than from a subpath of the base host.

Use <feature>, not {{feature}}: Deployer resolves {{...}} placeholders as soon as the configuration value is read, which would substitute the currently initialized instance everywhere the pattern is read, including for other instances (feature:list, the index page). <feature> is substituted explicitly instead.

Combining this with the url shortener (feature_url_shortener, see above) is recommended but not required: the shortener only changes where the instance is stored (.fbd/instances/<feature> symlinked from a flat directory), which either can serve as the subdomain's document root. See Web server for the required wildcard DNS entry, wildcard certificate and vhost configuration.

Because instance names become a DNS hostname label in this mode (see the table under Initialization), branch names that differ only by casing or by ./_ punctuation resolve to the same instance, and a name longer than 63 characters is truncated with a hash suffix.

Migrating an existing host: switching feature_url_pattern on for a host that already has path-mode instances changes their hostname-safe name (e.g. TEST-01 becomes test-01), which would otherwise orphan the existing directory while creating an empty new one under the new name. To guard against that, feature:init/feature:setup/feature:stop fail with an error if a directory under the pre-switch name still exists. Remove existing instances first against the previous configuration (feature:stop or feature:cleanup), then enable subdomain mode.

If you use feature:index, re-run it (feature:index) after enabling subdomain mode: an already-deployed index configuration predates the new featureUrlPattern key and needs to be re-rendered for the index page to link to the subdomains. Set feature_index_app_path relative to the application root (e.g. ''), not to current/public/, since there is no path prefix to traverse in this mode.

Scheduler

The feature:scheduler command uploads a standalone bash script to the target server, which runs the TYPO3 scheduler for all active feature branch instances. This is useful for executing periodic tasks (e.g. search indexing, mail queue processing) across all feature branches.

$ vendor/bin/dep feature:scheduler stage

The script is uploaded to the .fbd/ directory on the target server:

├── .fbd/
│   ├── scheduler.sh
│   ├── scheduler.log
│   └── ...

On the target server, add a crontab entry to run the scheduler periodically:

*/5 * * * * /var/www/html/.fbd/scheduler.sh >> /var/www/html/.fbd/scheduler.log 2>&1

The script requires no additional dependencies on the target system beyond what is already present for running TYPO3 (bash, PHP CLI). The paths for the PHP binary and the TYPO3 CLI are resolved from the deployer configuration (bin/php, bin/typo3cms) during upload.

The script automatically discovers all feature branch instances, skips instances without a current release and continues execution if the scheduler fails for a single instance.

Cleanup

The feature:cleanup command helps to cleanup feature branch instances. It compares remote git branches with remote feature instances and provides a cleanup for all untracked feature instances on the remote server.

$  vendor/bin/dep feature:cleanup
task feature:cleanup
+------------------- stage -------------------+
| Remote Git Branch | Remote Feature Instance |
+-------------------+-------------------------+
| main              | main                    |
| stage             | stage                   |
|                   | test                    |
+-------------------+-------------------------+
[stage] Do you want to cleanup all remote feature instances which haven't an according git branch anymore? (marked as red) [y/N] y
[stage] info feature branch test deleted

The remote branches are queried with git ls-remote --heads origin, so the task does not depend on the local checkout and works in CI with a detached HEAD or a shallow clone. If no branch is returned (e.g. the remote is unreachable), the task aborts instead of treating every instance as untracked.

To run the cleanup unattended, pass --force-cleanup. It skips both the overall and the per-instance confirmation, which would otherwise default to "no" in a non-interactive run and delete nothing. A nightly pipeline schedule in GitLab CI keeps a simple database pool from filling up with instances of merged branches:

feature-cleanup:
  stage: deploy
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule" && $FEATURE_CLEANUP == "1"
  script:
    - vendor/bin/dep feature:cleanup stage --force-cleanup

The job needs read access to the git remote and the same SSH access to the target host as a regular deployment.

Further more

Additional configurations regarding the feature branch deployment are available here: set.php

A bunch of helpful global deployer functions can be found here: set.php

FAQ

  1. Debugging

    For extended debugging information use the environment variable DEPLOYER_CONFIG_VERBOSE.

    Disable the notifications with the environment variable DEPLOYER_CONFIG_NOTIFICATION_MUTE.