Troubleshooting

Preliminaries

This is a high performance tool, and like all such engineering efforts, you can get yourself into trouble when you try out new features. Here are some best practices to follow as you build out your server.

Port 443

If you are unable to reach the server on port 443, check your firewall settings. If you are able to use a high port like 8443, but not 443, execute this command:

setcap 'cap_net_bind_service=+ep' /usr/bin/node
						

SSL Certificates

When using a hostname alias, remember that it will only work if you have a "wildcard" SSL certificate, which includes both the hostname and its aliases. Examine your certificate's "Certificate Subject Alt Name" section.

cURL will refuse to connect with a self-signed certificate. Adding an --insecure or -k option to the command line will override this.

File permissions

If the logger reports a file permission violation, review the correct settings for ownership and permissions at File Permissions.

Node.js® location

By default, the Linux package managers (dnf for Fedora/CentOS/RedHat and apt for Ubuntu/Debian) install the node executable to /usr/bin/node. Standard RWSERVE installations are compatible with this.

If you install a newer version of node using nvm or n, they will be installed to /usr/local/bin/node instead, and you will need to edit the /usr/lib/systemd/system/rwserve.service configuration to point to that location.

[Service]
ExecStart=/usr/local/bin/node /usr/lib/node_modules/rwserve/dist/main.js /etc/rwserve/rwserve.conf
Restart=no
User=rwserve
Group=rwserve 
WorkingDirectory=/srv
StandardOutput=journal
StandardError=journal
						

Be sure to save the file with Linux line endings (LF) not Windows line endings (CRLF). Then, issue these commands to reload systemd, grant network capabilities, and restart the HTTP/2 server:

systemctl daemon-reload
setcap 'cap_net_bind_service=+ep' /usr/local/bin/node
systemctl restart rwserve            
						

Registration

Failure to supply a valid customer number and access key will be reported in the logger upon startup. See Registration for how to specify these. Repeated failures will eventually disable the server.

Simultaneously using the same access key on more than one server will be reported in the logger upon startup. (Transferring the access key from one device to another is allowed and will not be reported, when not abused.)

Licenses have a limited lifetime. Notices are issued 15 days before expiration. An additional 7 day grace period is allowed for continued operation under an expired license.

When fixing customer-number and access-key issues, allow 1 hour to lapse before trying again.

Invalid customer-numbers and invalid access-keys are treated as forgeries which prevent the server from starting.

Registration settings are verified the first time the server is started, and thereafter on an infrequent basis, whenever checking for outdated versions or the availability of security updates.

Config Syntax

See Configuration Syntax for details.

These mistakes are common:

• Always use GRAVE-ACCENTS to delimit filenames and paths.
• Attribute names always begin with an ASTERISK.
• Always use APOSTROPHES or QUOTATION-MARKS to delimit configuration attribute values that contain spaces or that begin with an initial ASTERISK or FULL-STOP.

Check that all sections and subsections of the configuration file have matched opening and closing curly braces. Only the "One True Brace" syntax is allowed; "Allman style" braces are not allowed.

Remember that host settings are merged with server settings by default. See Merged Configurations for how these rules are implemented.

Carefully check that subsections are inside the correct section. See Config Hierarchy for quick reference.

Use the command line --verify option to check your configuration for possible problems before restarting the server with systemctl.

Modules

The modules configuration section is a master switch, and should always be checked when diagnosing why the server is not responding as expected. Two common mistakes are:

• having the module turned on, but forgetting to add its configuration rules;
• having a module's section correctly configured, but neglecting to turn the module on.

Logging

The Logging facility is your friend. When troubleshooting request/response problems adjust what's logged to provide greater visibility into the request/response cycle.

• Always examine the log after startup to look for unusual "config" entries.
• To better understand requests that don't respond as expected, enable "staging" to see processed WorkOrder data.
• When you can't understand why a response code is not 200, enable "information" header logging.
• To see TLS rejections, enable "'network" logging.
• To monitor cluster worker startup/shutdown, enable "cluster" logging.
• Enable "debug" logging, and add debug statements to your plugin with log.debug(method, message) to trace the flow of your plugin's execution.

Use the command journalctl -u rwserve -f to continuously monitor the request/response cycle.

Developing plugins

Use cURL 7.47.0 or later to automatically issue HTTP/2 requests; specify --http2 for earlier versions.

When using the Node.js inspector, configure multiple ports starting at 9229 for the master worker and 9230, 9231, ... for the workers. Add 'debugger;' statements to your plugin code to halt execution at targeted breakpoints. For debugging convenience, limit the cluster-size to 1 or 2, to reduce the confusion surrounding breakpoints.

Review

Best practices:

• Use a private git repository (or another version control system) to track changes to your configuration settings.
• Turn on logging of "staging" and "information" during initial testing.
• Limit cluster size when using Node.js inspector while debugging plugins.