Alerts

HTTP(s) clone access has been restored: The issue was narrowed down to a stale nginx configuration which directed authorization requests to the wrong port. The configuration has been corrected and normal service is restored. (08:35 UTC — Oct 27)

HTTP(s) clone is currently unavailable for hg.sr.ht: A planned deployment ran afoul and has caused some problems with cloning hg.sr.ht repositories via https. An investigation is ongoing. SSH clone is still working correctly. (08:20 UTC — Oct 27)

Planned maintenance is complete: We have finished upgrading all of the affected hosts and all services are restored to normal. (14:20 UTC — Aug 16)

Planned maintenance is ongoing: Our maintenance window has opened and we have started our work. (13:00 UTC — Aug 16)

Planned maintenance on August 16th will cause intermittent outages: We are planning the second (and last phase) of the maintenance which began on August 3rd. This will affect all services, causing intermittent outages that are expected to last between 15 and 30 minutes at most. The total maintenance period should last less than 2 hours. (13:00 UTC — Aug 16)

Maintenance complete: The issue with pages.sr.ht has been resolved and all services are now available. (10:30 UTC — Aug 3)

Maintenance mostly complete but pages.sr.ht still pending: We have completed maintenance on all services, which can now be expected to be stable, with the exception of pages.sr.ht. We have encountered an issue during the pages.sr.ht upgrades and are addressing it now. (10:00 UTC — Aug 3)

Planned maintenance on August 3rd will cause intermittent outages: Planned maintenance will affect all services, causing intermittent outages that are expected to last between 15 and 30 minutes at most. The total maintenance period should last less than 2 hours. (09:00 UTC — Aug 3)

Snapshot growth caused an outage on git.sr.ht

git.sr.ht’s ZFS snapshots grew to consume all available disk space. This is normally an understood pathology of the server configuration, but due to a change in billing with Twilio, our paging script did not alert the operators to the imminent issue. It seems that there is not a grace period with Twilio; they reported the billing issue to us only yesterday.

The issue with git.sr.ht has been resolved, the bill has been paid, and we are researching ways to avoid this occuring again in the future. (15:00 UTC — May 14)

Planned maintenance has been completed. (16:30 UTC — Feb 8)

Planned maintenance is now underway. (15:00 UTC — Feb 8)

Planned maintenance on February 8th will cause intermittent outages: Planned maintenance will affect all services, causing intermittent outages that are expected to last between 15 and 30 minutes at most. (15:00 UTC — Feb 8)

One of our third-party DNSBL services, SpamCop, allowed their domain to expire, presumably as a mistake, and began to return “listed” for all DNSBL checks. We use a DNSBL as an early rejection for spam emails, and this caused 21 incoming emails to be wrongfully rejected. We have removed SpamCop from our list of DNSBLs and filed a ticket to improve our monitoring so that we may catch this sooner. Incoming emails are working now, but be advised that if your postmaster uses SpamCop, emails from Sourcehut will likely be rejected until the issue is resolved. (13:25 UTC — Jan 31)

An issue with mail delivery caused some emails to be dropped. Upgrades to our mail server caused authentication to unexpectedly and silently fail, and it took some time to detect. During this period, ticket submission to todo.sr.ht failed, and emails were not forwarded from lists.sr.ht. You can thank Oracle for changing the license terms on Berkely DB, which had wider reaching consequences than we expected. (19:00 UTC — Jan 21)

An issue with our upstream ISP caused an outage. Our ISP decided to ring in the new year by unplugging us for 12 minutes last night. Sorry about that. (09:03 UTC — Jan 8)

The issued was addressed without disruption. While investigating the cause of disk growth, an error with our ZFS snapshot retention policy was found to be the cause. The disk space was trivially reclaimed without incurring an outage. (15:50 UTC — Oct 8)

Advances in the pace of disk usage on git.sr.ht has increased the urgency of a planned migration. On September 18th, we were tracking the growth of git.sr.ht disk space usage up to an expected disk space exhaustion in November. However, in the intervening time, the rate of growth has accellerated and we are now urgently seeking to migrate to a server we have prepped with more storage space. This may cause disruptions as git.sr.ht may become read-only during the migration process. (15:00 UTC — Oct 8)

Maintenance complete. Depending on how soon your DNS server picks up the updates, you should find service restored shortly. (21:00 UTC — Jul 29)

We’re still waiting on the latest data to transfer from the old server to the new. It is taking longer than we expected. The process is about halfway complete. We apologise for the delay. (19:00 UTC — Jul 29)

Maintenance may require more time than expected. (17:50 UTC — Jul 29)