Application Container Platform Support Portal Documentation
Let's Encrypt Rate Limiting
Investigating - We're currently aware of an ongoing issue in Not-Prod and Prod where tenants may be receiving an error when trying to deploy a new ingress resource. This is due to an issue with our current certificate issuer - Let's Encrypt - where we are regularly exceeding the rate limit for issuing new certificates.

Please note that the rate limit only impacts issuing NEW SSL certificates. It does not impact the renewal of already existing certificates.

To mitigate this issue we have started to deploy a new certificate issuer to our clusters. The new issuer has been deployed to the Not-Prod cluster and will be deployed to the Production cluster on Tuesday, April 9th.

If you're currently experiencing an issue with the creation of new certificates whilst deploying a new ingress, please raise an incident via the ACP Support Desk and one of our engineers will work with you to rectify the problem.

We are aiming to send out guidance to the wider ACP community on how to use the new issuer once we've got it deployed to all of the clusters.
Apr 05, 2024 - 15:23 BST
Uptime over the past 90 days. View historical uptime.
Artifactory ? Operational
90 days ago
100.0 % uptime
Today
Drone Github ? Operational
90 days ago
100.0 % uptime
Today
Drone Gitlab ? Operational
90 days ago
100.0 % uptime
Today
Gitlab ? Operational
90 days ago
100.0 % uptime
Today
Jira Service Desk ? Operational
90 days ago
100.0 % uptime
Today
Keycloak ? Operational
90 days ago
99.98 % uptime
Today
Kubernetes (not-prod) ? Degraded Performance
90 days ago
100.0 % uptime
Today
Kubernetes (prod) ? Operational
90 days ago
100.0 % uptime
Today
Platform Hub ? Operational
90 days ago
100.0 % uptime
Today
Sonarqube ? Operational
90 days ago
100.0 % uptime
Today
Sysdig ? Operational
90 days ago
99.97 % uptime
Today
VPN ? Operational
90 days ago
100.0 % uptime
Today
OpenSearch NotProd ? Operational
90 days ago
100.0 % uptime
Today
OpenSearch Prod ? Operational
90 days ago
100.0 % uptime
Today
OpenSearch Ops ? Operational
90 days ago
100.0 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.

No incidents or maintenance related to this downtime.

Scheduled Maintenance
Gitlab Production Upgrade (Thursday 4th April 18:00) Apr 4, 2024 18:00-22:00 BST
Gitlab Production Upgrade (Thursday 4th April 18:00)
Posted on Mar 28, 2024 - 16:39 GMT
Gitlab Production Upgrade (Thursday 4th April 18:00) Apr 4, 2024 18:00-22:00 BST
Gitlab Production Upgrade (Thursday 4th April 18:00)
Posted on Mar 28, 2024 - 16:39 GMT
Past Incidents
Apr 19, 2024

No incidents reported today.

Apr 18, 2024

No incidents reported.

Apr 17, 2024

No incidents reported.

Apr 16, 2024

No incidents reported.

Apr 15, 2024
Notprod Keycloak (v15) Outage
Resolved - This incident has been resolved.
Apr 15, 16:54 BST
Monitoring - The issues with the notprod Keycloak instance (v15) have now been resolved - please do sign in and let us know if you experience any further issues. We have now disabled the realm which was producing the large number of requests and made some adjustments to the database configuration. We will be monitoring the database closely to ensure this issue does not occur again. Apologies for the disruption caused.
Apr 15, 10:50 BST
Investigating - ACP are aware of issues with the notprod Keycloak instance (v15) which have been resulting in access issues for multiple tenants since Friday late afternoon. We are working on resolving this as soon as possible and will update here once the solution has been applied and log in is working as expected. It appears this issue has arisen due to a significant increase in requests to the database on Friday. We will be investigating further and will communicate root cause once confirmed.
Apr 15, 09:57 BST
Apr 14, 2024

No incidents reported.

Apr 13, 2024

No incidents reported.

Apr 12, 2024

No incidents reported.

Apr 11, 2024

No incidents reported.

Apr 10, 2024

No incidents reported.

Apr 9, 2024
ACP Prod Cluster Issuer deployment - Tuesday, April 9th
Completed - ACP will be deploying a new certificate issuer to the ACP Prod cluster to remediate an ongoing rate limiting issue with our current certificate issuer where we are regularly exceeding the limit for creation of new SSL Certificates. This limit does not impact the renewal of already existing certificates, only the creation of new certificates.
Apr 9, 14:19 BST
In progress - ACP will be deploying a new certificate issuer to the ACP Prod cluster to remediate an ongoing rate limiting issue with our current certificate issuer where we are regularly exceeding the limit for creation of new SSL Certificates. This limit does not impact the renewal of already existing certificates, only the creation of new certificates.
Apr 9, 14:02 BST
Scheduled - ACP will be deploying a new certificate issuer to the ACP Prod cluster to remediate an ongoing rate limiting issue with our current certificate issuer where we are regularly exceeding the limit for creation of new SSL Certificates. This limit does not impact the renewal of already existing certificates, only the creation of new certificates.
Apr 5, 14:09 BST
Apr 8, 2024

No incidents reported.

Apr 7, 2024

No incidents reported.

Apr 6, 2024

No incidents reported.

Apr 5, 2024
Apply new Sysdig License
Completed - As the Sysdig-Prod license is approaching time of expiration, we will need to apply the new license to continue to utilise the service.

This necessitates a very brief service interruption, as per the Sysdig documentation we require to restart some of the Sysdig Backend Components (however these are controlled by multi-replica deployments w/ failover, so impact should be minimal)

Given the short time-frame until expiration, this will be put forward as an Emergency Change (as there is less than 1/2 weeks to give notice)
Apr 5, 16:34 BST
Update - As the Sysdig-Prod license is approaching time of expiration, we will need to apply the new license to continue to utilise the service.

This necessitates a very brief service interruption, as per the Sysdig documentation we require to restart some of the Sysdig Backend Components (however these are controlled by multi-replica deployments w/ failover, so impact should be minimal)

Given the short time-frame until expiration, this will be put forward as an Emergency Change (as there is less than 1/2 weeks to give notice)
Apr 5, 16:33 BST
In progress - As the Sysdig-Prod license is approaching time of expiration, we will need to apply the new license to continue to utilise the service.

This necessitates a very brief service interruption, as per the Sysdig documentation we require to restart some of the Sysdig Backend Components (however these are controlled by multi-replica deployments w/ failover, so impact should be minimal)

Given the short time-frame until expiration, this will be put forward as an Emergency Change (as there is less than 1/2 weeks to give notice)
Apr 5, 16:01 BST
Scheduled - As the Sysdig-Prod license is approaching time of expiration, we will need to apply the new license to continue to utilise the service.

This necessitates a very brief service interruption, as per the Sysdig documentation we require to restart some of the Sysdig Backend Components (however these are controlled by multi-replica deployments w/ failover, so impact should be minimal)

Given the short time-frame until expiration, this will be put forward as an Emergency Change (as there is less than 1/2 weeks to give notice)
Apr 3, 17:18 BST
← Incident History Powered by Atlassian Statuspage