Scyld Cloud Accountant

An SCM cluster has resources that are consumable by its users, such as allocated storage and core-hours consumed on a compute node. Resource usage is stored in a dedicated database and the billable usage can typically be summarized in daily units over a single metric (such as core-hours, GB, or time). The Scyld Cloud Accountant periodically polls these resource databases, calculates the single usage metric for that particular resource type, and stores it within the accountant database. Users and administrators can then retrieve daily resource usage summaries from the accountant and retrieve results in JavaScript Object Notation (JSON) format using HTTP/HTTPS requests.

Resource Families and Resource Types

Resource types are grouped into resource families if they describe similar resources. Each resource family is expected to have a specific summary metric. Three different resource families are currently supported by the Scyld Cloud Accountant:

  • Storage resource usage is defined by the amount of disk bytes allocated.
  • Server-Instance resource usage is defined by the amount of time that a server has been used.
  • Job resource usage is defined by the number of core-hours used across all jobs.

While there is currently only one generic type of Storage and Server-Instance supported by the Scyld Cloud Accountant, there are several types of Job resource types, including:

  • PBS Torque
  • SGE
  • UGE
  • LSTC

In addition to daily summary information, the Scyld Cloud Accountant also has the ability to provide more detailed usage information about PBS Torque, SGE, and UGE jobs.

Access Permissions

Access to daily usage summaries is determined by Cloud Auth. While every user is allowed to see their own resource usage information, account owners are also able to query the accountant for resource usage information on their managed users.

Scyld Cloud Accountant Setup

The cloudaccountant must be configured to connect to each cloudcontroller instance in the SCM installation. In most cases, there is a single cloudcontroller. Each cloudcontroller instance has a unique id: the first one is id 1, the next is id 2, etc.

  • sqlalchemy.cloud-controller-id.resource-type.url - Specifies the database URL for the resource-type for a given cloud-container-id. Currently assumes MySQL databases. For example: To point to the server-instances resource-type of cloud-controller-id 1, one would set sqlalchemy.cloud.1.instances.url = url

Initializing the Scyld Accountant Database

Note

This only applies to MySQL databases that have not been initialized.

If the Scyld Cloud Accountant database is ever destroyed and needs to be recreated, you can do so by running the database initialization script. Begin by logging in as root and activating the python environment for the accountant:

Verify that sqlalchemy.accountant.url has been properly defined in cloudaccountant.ini.

Now run the initialization script:

# source /var/www/wsgi/cloudaccountant/env/bin/activate
# initialize_scyld_cloud_accountant_db/var/www/wsgi/cloudaccountant/scyld-cloud-accountant/cloudaccountant.ini

Importing from Resource Databases

To manually import data from the resource databases into the cloud accountant databases, first, activate the Python environment for the accountant, then run the import script:

# source /var/www/wsgi/cloudaccountant/env/bin/activate
# scyld_cloud_accountant_db_import /var/www/wsgi/cloudaccountant/scyld-cloud-accountant/cloudaccountant.ini <START DATE> <END DATE>

This will import data relevant to all resources with URLs listed in the cloudaccountant.ini file between the given <START DATE> and <END DATE>.

NOTE: Importing over the same dates repeatedly will not remove any existing entries or produce duplicate rows. Importing will only add or modify existing entries.

To have this operation performed regularly, this action is usually handled as part of a cron script.

Clearing Data from the Scyld Cloud Accountant Database

To manually import data from the resource databases into the cloud accountant databases, first activate the Python environment for the accountant, then run the import script:

# source /var/www/wsgi/cloudaccountant/env/bin/activate
# scyld_cloud_accountant_db_import /var/www/wsgi/cloudaccountant/scyld-cloud-accountant/cloudaccountant.ini start_date end_date

This will import data relevant to all resources with URLs listed in the cloudaccountant.ini file between the given start_date and end_date.

Note

Importing over the same dates repeatedly will not remove any existing entries or produce duplicate rows.

Importing will only add or modify existing entries.

To have this operation performed regularly, this action is usually handled as part of a cron script.