Become a premium member to remove ads
AwakenedRage

Service Issues 2016

6 posts in this topic

Hello Everyone,

 

This is Michael, and I regret to inform you all that last Tuesday we were notified of a potential exploit in MySQL based database servers. Since I am the Database Administrator for this community, it was my job to prepare for such an instance and immediately I took the proper steps to make sure our servers were not vulnerable to this exploit. At the time we currently had eight servers in different datacenters across the United States and Canada using a replication setup to prevent downtime. After checking all of our servers, we were able to determine that 1 out of the eight servers had the exploit. This specific server is for our web hosting for the forums, and this particular server I have no control over. The past couple days have mainly consisted of Halo and me making necessary measures to keep the site secure while we waited for a response from Host1Plus. Our web hosting provider has put us in this situation where the database server provided by them is using a build all the way back from 2014. This reckless action potentially makes the site vulnerable to many exploits. It is with that decision that we have made efforts to configure the website to use a Database Server outside of Host1Plus until this issue is resolved. However, at this time, we recommend that you change your password to the forums and also any passwords that were the same as the one used on here. I am not issuing mandatory password resets, however, if it comes to the point where Host1Plus admits to making a mistake, then I will be sure to issue mandatory password resets ASAP.

 

This is the reason why on Saturday, Sunday, and Monday that the Forums had some downtime regarding connection to the Database. There may be a bit of a delay on the site too because the Database server takes a couple seconds to send the information needed to load the page. 

 

If I get anymore updates from Host1Plus I will be sure to keep you all posted. 

 

Sincerely,
Michael/AwakenedRage
SIGMA Head Developer
CU Database Administrator

Share this post


Link to post
Share on other sites

Okay, so here's an update for you guys. Host1Plus has since then upgraded the server to a more up to date version. I am confirming with them right now the CVE's in future releases are patched and accounted for. I am also looking into some other potential risks to the web server.

Share this post


Link to post
Share on other sites

13 hours ago, AwakenedRage said:

Okay, so here's an update for you guys. Host1Plus has since then upgraded the server to a more up to date version. I am confirming with them right now the CVE's in future releases are patched and accounted for. I am also looking into some other potential risks to the web server.

 

If so then we can move the databases back to the webhost, so the load times aren't miserable. :)

Share this post


Link to post
Share on other sites

Unfortunately no,

 

Its still has some concerns present. Thankfully, though we only have one CVE left out of many that were discussed back and forth. As for the bad load times, I am not having those issues. 

Share this post


Link to post
Share on other sites

As some of you probably noticed that the database was rolled, back. We had an Chaotic failure on all four of our database servers not being in sync with each other. 

I managed to restore from a backup I took this afternoon through the hosts panel. However, the other servers, mainly Masters 2, and Slave 1 are offline. This means for the next 16 hours we will not be using a replication setup because I simply want the forums to be up as soon as possible. 

 

I have constantly been making improvements to our database system, and in order to prevent this from happening again in the future, I have decided to host all the databases with one company. This means that the Slave will be terminated immediately and Master 2 will be pending for deletion. We will have three new VPS's hopefully tomorrow. 

 

Steps Taken:

- Slave from DigitalOcean deleted

 

Heres what steps I will be taking tomorrow.

- 3 VPS's will be ordered tomorrow

- These new servers will be configured. 

- The Large Databases will be imported, before the Website's databases.

-

THEN ONLY WILL the forum being taken down for maintenance for a short time before it will be back up.

 

These measures are being put in place to prevent future downtime,

 

 

 

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now