Technical Summary

The configuration for jupyterhub-usage-quotas include the following requirements:

• Admins can declaratively define quota limits over a rolling window^[1]

• Quota limits are defined at the user group-level and individually applied to members on a per-hub basis

• Prometheus provides a single source of truth for usage metrics

• Consistent usage and quota information can be re-used, e.g. consumed by a user-facing JupyterHub service.

Input¶

We apply the design principle of separating mechanism from policy. We avoid mixing how the system may apply quotas (mechanism) with who and what quotas may apply to (policy).

See UsageQuotaConfig for all possible configuration options.

Policy configuration¶

Compute quotas¶

A quota policy for compute can be defined as

c.UsageQuotaManager.policy = [
  {
    "resource": "memory" | "cpu", str,
    "limit": int | str,
    "window": int,
    "scope": {
        "group": [str],
        "user": [str],
        "service": [str],
    }
  }
]

This accepts what resource the quota applies to, its limit value, and the time period for the rolling window. The scope of who the quota applies to mirrors the JupyterHub RBAC framework.

Fallback strategy¶

c.UsageQuotaManager.scope_fallback_strategy is used to set a quota resolution strategy in the case where the scope of the quota policies cover no users, or applies multiple policies to a single user. In the case where no quota is applied, we can supply a default quota policy or leave this empty for unlimited quotas; and where multiple quotas are applied, we can apply operators min, max or sum to the limit.

Mechanism configuration¶

This describes how the quota system is applied. For example, we can configure the usage metric to query Prometheus with using memory requests by supplying

c.UsageQuotaManager.prometheus_usage_metrics = {
    "memory": "kube_pod_container_resource_requests{resource='memory'}",
}

See kubernetes/kube-state-metrics for details on the metrics exported by Kubernetes.

Another example of configuring the quota system mechanism is the case where the quota system is unavailable. We can set c.UsageQuotaManager.failover_open to True to allow all server launches with no restriction when the quota system is offline, or to False to deny all server launches to prevent unaccounted compute usage.

Decision Logic¶

In this explanation, we constrain compute usage by memory requests to a rolling window of 30 days to demonstrate the decision logic of the quota system.

Usage metrics¶

kube-state-metrics exports the metric kube_pod_container_resource_requests^[2], which measures the amount of compute resources requested by the Kubernetes scheduler in bytes. We multiply this by c.UsageQuotaManager.prometheus_scrape_interval and divide by 60² to convert from resource-seconds per sample to resource-hours.

Policy resolver¶

The policy resolver matches users with policy scopes to determine the quota limit applied. When a user is a member of multiple groups, the configured strategy from c.UsageQuotaManager.scope_fallback_strategy["intersection"] is applied. When a user is not a member of any group, the system can be configured to default to no quota limits or quota limits specified in c.UsageQuotaManager.scope_fallback_strategy[“empty”]. In the case where multiple quota policies apply over different rolling windows, then each policy is returned and applied with no limit stacking.

Metric aggregation¶

Prometheus collects metric samples on a regular basis that can be passed to c.UsageQuotaManager.scrape_interval. If the scrape interval is 60 seconds, then over 30 days there will be 43,200 samples.

To calculate usage over the last 30 day window, we need to integrate over time for a result independent of the sample granularity:

sum(sum_over_time(kube_pod_container_resource_requests{resource="memory|cpu"}[30d])) * scrape_interval

We divide the result by 60 * 60 to convert to hours.

The result^[3] of the above PromQL pseudo-query is then compared against the each policy quota limit returned by the policy resolver. We use pure values, i.e. bytes-hours and cpu-hours, in this comparison.

If the result is less than the policy limit, then we return output["allow_server_launch"]=True. If the result is greater than the policy limit then output["allow_server_launch"]=False and a structured error is processed and returned.

Retry time¶

With a rolling window, quota expires continuously over time. When a user exceeds their quota, a useful output to calculate is the retry_time so that users know when resources are available again. This is calculated by:

1. finding the difference between the usage at the current time, $t_c$, and the quota limit, $\Delta r$

2. finding the historic timestamp within the rolling window where $\Delta r$ has been used, $t_e$,

3. projecting the future timestamp with adding the rolling window period, $t_w$, to $t_e$ to determine the retry time, $t_r$.

See Figure 1 for a graphical example.

Output¶

The output of the jupyterhub-usage-quotas system is structured as:

output = {
  "allow_server_launch": True | False,
  "error": None | {
    "code": str,
    "message": str,
    "retry_time": str | None
  }
  "quota": {
    "resource": str,
    "pure_used": float,
    "used": float,
    "pure_limit": float,
    "limit": str,
    "unit": str,
    "readable_unit": str,
    "window": int,
    "scope": {
      "group": [str],
      },
  },
  "timestamp": str,
}

This can be consumed by kubespawner with a pre-spawn hook that will launch a server if allow_server_launch is True. If allow_server_launch is False, then a response can present information from error to the user to explain why a server launch was denied.