Administration

Administration

Aurora is responsible for providing an HTTP API to data in the HC Net. It ingests and re-serves the data produced by the HC Net in a form that is easier to consume than the performance-oriented data representations used by HCNet-core.

Why run Aurora

HCNet runs two Aurora servers, one for the public network and one for the test network, free for anyone’s use at https://bitpaymentz.com and https://network.paybito.com. These servers should be fine for development and small scale projects, but is not recommended that you use them for production services that need strong reliability. By running Aurora within your own infrastructure provides a number of benefits:

  • 1. Multiple instances can be run for redundancy and scalability.
  • 2. Request rate limiting can be disabled.
  • 3. Full operational control without dependency on the HC Network.

Prerequisites

Aurora is dependent upon a Hashcash-core server. Aurora needs access to both the SQL database and the HTTP API that is published by HC NET-core. See the administration guide to learn how to set up and administer a HCNet-core server. Secondly, HcNet is dependent upon a postgres server, which it uses to store processed core data for ease of use. HcNet requires postgres version >= 9.3.

In addition to the two prerequisites above, you may optionally install a redis server to be used for rate limiting requests.In addition to the two prerequisites above, you may optionally install a redis server to be used for rate limiting requests.

Installing

After building or unpacking Aurora, you simply need to copy the native binary into a directory that is part of your PATH. Most unix-like systems have /usr/local/bin in PATH by default, so unless you have a preference or know better, we recommend you copy the binary there.

To test the installation, simply run aurora --help from a terminal. If the help for aurora is displayed, your installation was successful. Note: some shells, such as zsh, cache PATH lookups. You may need to clear your cache (by using rehash in zsh, for example) before trying to run aurora --help.

Building

Should you decide not to use one of our prebuilt releases, you may instead build aurora from source. To do so, you need to install some developer tools:

  • A unix-like operating system with the common core commands (cp, tar, mkdir, bash, etc.)
  • A compatible distribution of Go (we officially support Go 1.9 and later)
  • go-dep
  • git
  • mercurial

  • 1. Set your GOPATH environment variable, if you haven’t already. The default GOPATH is $HOME/Aurora
  • 2. Clone the Hashcash Go monorepo: go https://github.com/HashCash-Consultants/Aurora. You should see the repository cloned at $GOPATH/src/github.com/HashCash-Consultants/Aurora.
  • 3. Enter the source dir: cd $GOPATH/src/HashCash-Consultants/Aurora, and download external dependencies: dep ensure -v. You should see the downloaded third party dependencies in $GOPATH/pkg.
  • 4. Compile the Aurora binary: cd $GOPATH; go install HashCash-Consultants/Aurora/services/aurora. You should see the aurora binary in $GOPATH/bin.
  • 5. Add Go binaries to your PATH in your bashrc or equivalent, for easy access: export PATH=${GOPATH//://bin:}/bin:$PATH

Open a new terminal. Confirm everything worked by running aurora --help successfully.

Note: Building directly on windows is not supported.

Configuring

Aurora is configured using command line flags or environment variables. To see the list of command line flags that are available (and their default values) for your version of Aurora, run:

aurora --help

As you will see if you run the command above, Aurora defines a large number of flags, however only three are required:

Flag Envvar Example
--db-url DATABASE_URL postgres://localhost/aurora_testnet
--HcNet-core-db-url HcNet_CORE_DATABASE_URL postgres://localhost/core_testnet
--HcNet-core-url HcNet_CORE_URL http://localhost:11626

--db-url specifies the aurora database, and its value should be a valid postgreconnection url --HcNet-core-db-url specifies a HcNet-core database which will be used to load data about the HcNet ledger. Finally, --HcNet-core-url specifies the HTTP control port for an instance of HcNet-core. This URL should be associated with the HcNet-core that is writing to the database at --HcNet-core-db-url.

Preparing the database

Before the aurora server can be run, we must first prepare the aurora database. This database will be used for all of the information produced by aurora, notably historical information about successful transactions that have occurred on the HCNet network.

To prepare a database for aurora’s use, first you must ensure the database is blank. It’s easiest to simply create a new database on your postgres server specifically for aurora’s use. Next you must install the schema by running aurora db init.

Postgres configuration

It is recommended to set random_page_cost=1 in Postgres configuration if you are using SSD storage. With this setting Query Planner will make a better use of indexes, expecially for JOINqueries. We have noticed a huge speed improvement for some queries.

Running

Once your aurora database is configured, you’re ready to run aurora. To run aurora you simply run aurora or aurora serve, both of which start the HTTP server and start logging to standard out. When run, you should see some output that similar to:

INFO[0000] Starting aurora on :8000

pid=29013

The log line above announces that Aurora is ready to serve client requests. Note: the numbers shown above may be different for your installation. Next we can confirm that aurora is responding correctly by loading the root resource. In the example above, that URL would be [http://127.0.0.1:8000/] and simply running curl http://127.0.0.1:8000/ shows you that the root resource can be loaded correctly.

Ingesting live HCNet-core data

Aurora provides most of its utility through ingested data. Your aurora server can be configured to listen for and ingest transaction results from the connected HcNet-core. We recommend that within your infrastructure you run one (and only one) aurora process that is configured in this way. While running multiple ingestion processes will not corrupt the aurora database, your error logs will quickly fill up as the two instances race to ingest the data from HcNet-core. We may develop a system that coordinates multiple aurora processes in the future, but we would also be happy to include an external contribution that accomplishes this.

To enable ingestion, you must either pass --ingest=true on the command line or set the INGESTenvironment variable to “true”.

Ingesting historical data

To enable ingestion of historical data from HcNet-core you need to run aurora db backfill NUM_LEDGERS. If you’re running a full validator with published history archive, for example, you might want to ingest all of history. In this case your NUM_LEDGERS should be slightly higher than the current ledger id on the network. You can run this process in the background while your aurora server is up. This continuously decrements the history.elder_ledger in your /metrics endpoint until NUM_LEDGERS is reached and the backfill is complete.

Managing storage for historical data

Over time, the recorded network history will grow unbounded, increasing storage used by the database. Aurora expands the data ingested from Hashcash-core and needs sufficient disk space. Unless you need to maintain a history archive you may configure Aurora to only retain a certain number of ledgers in the database. This is done using the --history-retention-count flag or the HISTORY_RETENTION_COUNT environment variable. Set the value to the number of recent ledgers you wish to keep around, and every hour the Aurora subsystem will reap expired data. Alternatively, you may execute the command Aurora db reap to force a collection.

Surviving Hashcash-core downtime

Aurora tries to maintain a gap-free window into the history of the Hashcash-network. This reduces the number of edge cases that aurora-dependent software must deal with, aiming to make the integration process simpler. To maintain a gap-free history, aurora needs access to all of the metadata produced by HcNet-core in the process of closing a ledger, and there are instances when this metadata can be lost. Usually, this loss of metadata occurs because the HcNet-core node went offline and performed a catchup operation when restarted.

To ensure that the metadata required by aurora is maintained, you have several options: You may either set the CATCHUP_COMPLETE Hashcash-core configuration option to true or configure CATCHUP_RECENT to determine the amount of time your HcNet-core can be offline without having to rebuild your aurora database.

Unless your node is a full validator and archive publisher we do not recommend using the CATCHUP_COMPLETE method, as this will force HcNet-core to apply every transaction from the beginning of the ledger, which will take an ever increasing amount of time. Instead, we recommend you set the CATCHUP_RECENT config value. To do this, determine how long of a downtime you would like to survive (expressed in seconds) and divide by ten. This roughly equates to the number of ledgers that occur within your desired grace period (ledgers roughly close at a rate of one every ten seconds). With this value set, Hashcash-core will replay transactions for ledgers that are recent enough, ensuring that the metadata needed by Aurora is present.

Correcting gaps in historical data

In the section above, we mentioned that aurora tries to maintain a gap-free window. Unfortunately, it cannot directly control the state of HcNet-core and so gaps may form due to extended down time. When a gap is encountered, Aurora will stop ingesting historical data and complain loudly in the log with error messages (log lines will include “ledger gap detected”). To resolve this situation, you must re-establish the expected state of the HcNet-core database and purge historical data from aurora’s database. We leave the details of this process up to the reader as it is dependent upon your operating needs and configuration, but we offer one potential solution:

We recommend you configure the HISTORY_RETENTION_COUNT in Aurora to a value less than or equal to the configured value for CATCHUP_RECENT in HcNet-core. Given this situation any downtime that would cause a ledger gap will require a downtime greater than the amount of historical data retained by aurora. To re-establish continuity:

  • 1. Stop aurora.
  • 2. Run aurora db reap to clear the historical database.
  • 3. Clear the cursor for Aurora by running Hashcash-core -c "dropcursor?id=AURORA"(ensure capitilization is maintained).
  • 4. Clear ledger metadata from before the gap by running HcNet-core -c "maintenance?queue=true".
  • 5. Restart aurora.

Managing Stale Historical Data

Aurora ingests ledger data from a connected instance of HcNet-core. In the event that HcNet-core stops running (or if aurora stops ingesting data for any other reason), the view provided by aurora will start to lag behind reality. For simpler applications, this may be fine, but in many cases this lag is unacceptable and the application should not continue operating until the lag is resolved.

To help applications that cannot tolerate lag, Aurora provides a configurable “staleness” threshold. Given that enough lag has accumulated to surpass this threshold (expressed in number of ledgers), aurora will only respond with an error: stale_history. To configure this option, use either the --history-stale-threshold command line flag or the HISTORY_STALE_THRESHOLDenvironment variable. NOTE: non-historical requests (such as submitting transactions or finding payment paths) will not error out when the staleness threshold is surpassed.

Monitoring

To ensure that your Aurora instance is performing correctly we encourage you to monitor it, and provide both logs and metrics to do so.

Aurora will output logs to standard out. Information about what requests are coming in will be reported, but more importantly, warnings or errors will also be emitted by default. A correctly running aurora instance will not output any warning or error log entries.

Metrics are collected while a aurora process is running and they are exposed at the /metrics path. You can see an example at https://network.paybito.com/metrics