Twitter Updates
Please wait while my twitter updates load 
Please wait while my twitter updates load
Gentoo-Portage.com is up and running, time to get the wiki server up and a scratch wiki setup for testing settings.
This site now on what will be the live server.
http://www.gentoo-wiki.info/ - for a read-only version of many previous articles.
http://scratch.gentoo-wiki.com/ - for a test version of the wiki we are working on, feel free to mess around on it.
With the old Wiki gone, I intend to create a new version. Make it better then the first, fix the mistakes I made in the beginning. I will ned some dedicated editors help in the future, stay tuned.
for now if everyone can wait, the best you can do is donate a few dollars for some caffeinated beverages for myself and to help with any future costs.
Please note that I will eventually be setting up a discussion topic on forums.gentoo.org and setting up an irc channel for suggestions, help and so-forth.
At minimum 8 cores, 16 GB ram and 50 Gigs of space with a 10Mbit pipe (1.5 - 2Mbit average output). I have settled on going with a friend of the wiki Jeremy McSpadden's offer at this point.
I will be setting up VMware servers on this box Friday Oct 24th for both sites.
Being a simple setup this site should be back up Saturday morning, however lacking all previous user data (Sorry everyone).
At this point I will setup the forum post(s) and irc channel for input from the community. The wiki server software setup should take up most of my night on the Saturday, getting all the options, the plugins and the settings all perfect.
I will work to make setting up new language wikis quick and easy, and setup any for the former admins of those wikis. And have them follow the bellow steps as best and as fast as they can.
At this point I will call for new admins, and work on vetting them for the coming days. Admins will only need to monitor entries and make sure people keep things orderly nothing more. more to come on this when it's time.
People will be needed to re-write some of the plugins and bots that ran the wiki (and create a few new ones)... More Later.
With help from anyone willing to contribute we will define a new structure for the documents going onto the wiki. this time having a proper naming convention in place. and define any other rules people will think useful. (nothing too restrictive, I don't want to keep people from editing).
I will allow the first group of editors on to the wiki and open up public Read access to the site. The first editors will help write all forms of templates, wiki policies and admin-level documents.
While the first editors get on their way, I will setup a full backup procedure. 24 separate off-site backups. rsync of the file structure, and an encrypted mysql dump (privacy!) for both websites. backups to be made hourly. Est. backup need for those that apply (later please!) is 20 gigs storage, 50gb monthly transfer with a monthly increase of 1gb transfer.
Using Google Cache, Archive.org, and some text table backups from the march backup I will get a second wave of editors in to start restoring as much as possible.
Open up the Wiki to all for editing, hopefully bringing us back to where we were in a few short months. I hope to be at this point before December.
At approximately 1:00 AM October 17th both websites went down. heres what i know:
The scratch wiki has gotten a LOT of good ides, and tonight I will be bringing the final version of the wiki live. Upon opening everyone will be able to register, and setup their preferences, watch lists and stuff. To begin only I will be able to edit, after I get the main pages started I will start letting other people in to edit. If you wish to be one of the first editors, hop on the irc channel and I hope to start giving out tasks for people to do and to have an overall task list and timeline for the rest of the wiki. (I hope to allow public editing in 2 weeks)
Also I am looking for admins for other languages to begin the same process for other languages. You wont need to do too much editing yourself, just to keep everyone organized.
A scratch version of the wiki is setup where we will be able to test all the planned extensions and everything else. feel free to edit this wiki and test out any templates you' like to implement, articles you plan to restore later, or suggestions for extensions and anything else you want.
I have to head out for a while, but i'll be back to implement those extensions, setup the actual wiki for editing and everything else later tonight. If you plan on requesting to be an admin of the new wiki, please jump in IRC, if all goes well I'll be appointing the first set tonight. and later if all goes well on that end The first load of approved edits will be let in to start setting up templates, policy pages, and all other forms of help for when the wiki opens to live editing.
Thanks to everyone who will be helping out
Gentoo-portage.com IS BACK UP! it's ready to go and I encourage everyone to comment on packages and upload images like a crazy person while they wait for the wiki. I'll be setting up the basics of the wiki tomorrow morning, I'll setup the wiki IRC channel and the Forum post first thing and we can discuss with the community how we want to structure the new wiki.
Small syn flood.. dealing with it. Gentoo-Wiki Email is back up. mikevalstar@gentoo-wiki.com , if any former admins want their email address back give me a shout.
Well Skiplink got ahold of me again this morning, told me again this was TelXs fault. Granted he makes some very good points and he may be right. However I am still pissed that I was not informed of the outage for days, and that he refuses to help me retrieve the data via TeX as a proxy. Extortion it may be; but it would be a hell of a lot cheaper then the alternative.
*edit* Just thought I would summerize for everyone who dosent want to read all of the below stuff: everyone in this whole mess is at fault; I peronally failed to do proper backups, TelX was a dick about billing and has been extorting customers, And Skiplink handled the situation VERY poorly on the customer service end IMHO.
Starting on the VMs today, should be ready to put Gentoo-Portage up saturday morning no problem.
At this time I'm going to have to call the former site a loss. I'll post shortly on what will come next.
You guys can stop harassing Skiplink now, thanks for the sentiment though
If I were to sum up in one word why it has come to this it would be SPITE.
The full story: I have left out bits and pieces of the story so far because I did not want to sound mean to anyone who may or may not help me in the future and may be reading this (Hi Robert). However it has come to a point where I am on my own with little recourse; so I might as well tell everything I know. The story I have pieced together from people at TelX, emails form people in the building, people at 55 Marietta (Skiplink's new building) and other people who just know things.
It all began a while ago, TelX at 56 Marietta in Atlanta was overcharging Skiplink for space and hydro. Skiplink did what I would have done and decided to move (to 55 Marietta), to get the serves out quick. After moving a decent portion of his servers to the new facility TelX noticed and decided to extort some more money out of him before he left. Skiplink would of course not pay the extortion money so TelX shut off his cage and locked him out of the building.
At this point I come into the story and find my servers turned off. After several days of attempting to contact Skiplink via cell phone, office phone, text message, email, and IM my frustration grew. I had already restored most of my clients DNS and whatever backups I had available on to other servers and I was getting nowhere with Skiplink. Seeing as I was already starting to get a few emails from some resourceful googlers ad put up this temporary site. Emails rolled in with offers of support and help (thanks BTW) and a few emails with bits and pieces of information to tide me over.
Skiplink finally texted me with some minimal information, I basically took it as a "fuck you you're not important enough to warrant warning or details of what is going on". With that I at least hoped he was moving his servers, mine was just last on the list. Oh I was last on the list, but I wasent being moved; <Rant>I guess paying the equivalent of my mortgage in servers every month dosent make me important</Rant>. I received an email from some helpful users that gave me the information in the 1st paragraph so I called TelX.
TelX was somewhat helpful and was willing to accommodate me getting back my data (Skiplink would later call it extortion). They were willing to hook my machine up to a network and let me copy things off, or power up the entire cage and hope for the best. this would have cost me a bit of money but I and some of my clients would have gladly paid it. However i had gleaned at this point that Skiplink had already turned off the outside networks to the cage so powering it up would be useless. But I still had the option of hooking up the boxes directly, if i only knew which were mine.
After about a day and a half of pleading to Skiplink to give me something identifiable about the servers he refused saying "No they will not [power up the servers], we are in dispute with amounts owed". Adding "This has to be handled properly via appropriate channels." I asked further about getting some info about the servers and was told "Again it is all our equipment... You will have to wait on removal of your euipment". There was no more texting for a while and I got no more information out of skiplink for days.
This morning I finally chatted with Skiplink over IM and reiterated my request that he allow me to access my data via TelX as proxy. Skiplink was very adamant that the hardware was his and that TelX could not touch it. I informed him that they have said otherwise and would allow me access to the servers (for a fee) if i could determine which servers were mine. Skiplink went over the legal-ness of this with me, said TelX was extorting him and now me, telx has no right to hold his property, and in essence said that he would not help me. He also made it clear that he will not personally sue to get the equipment, or pay the fee of $25,000 TelX says he owes to retrieve the equipment. His last ditch effort was to send someone who had a collocated box on floor 2 to retrieve it and hope he was allowed onto floor 8 with my servers (and other peoples). I have not heard anything from him as of yet and can only assume he has failed.
And that brings us to today, October 23rd and exactly 1 week after the servers went down. To summarize my serves are in limbo, I have little legal recourse as I am in canada and don't actually own any physical property in this case. Skiplink will not tell me which servers are mine simply to spite TelX for wronging him. The wiki had no recent backups ( I have a backup of the content but not the users or other data from March thanks to a helpful user ). Gentoo-Portage I have a full code base backup, but none of the user content. and lastly I am exhausted from lack of sleep.
Final Rant: Skiplink is acting childish, has been completely unprofessional about the whole matter, and has been an ass about the whole thing. They should have told me what happened the day it happened, not a few text messages and a full explanation a full 1 week later. I will NEVER recommend their services to anyone, and will actively encourage current users to move their servers elsewhere.
Thanks to everyone who has emailed me over the past week, you have kept me from going completely insane and I thank you whole heartedly. The wiki will be back, it will be better stronger and most importantly properly backed up (my plan for moving forward will come in the next hour or so).
P.S. to the other people with servers stuck with mine and have emailed me. I feel for you and I will keep this space updated with as much info as I can.
Mike Valstar - Apologizing to everyone for not having off-site backups in place.
Well Finally had a real conversation with Skiplink today, according to him TelX has seized his property unlawfully due to lack of payment. he has a new datacenter location that I was one of the people that did not get transfered to before this all went down. from what I'm told he may be able to get my server back soon.
I'm currently wat work and don't want to go into too much detail at this point (I will go into detail later depending on the outcome of some events). but if Skiplink is unable to get the server back soon they will not be able to get them back for a long time (if ever). If I am unable to get the servers back I have a sorta-plan ready to go to get something
I will post again before tomorrow. stay tuned
Where We Stand:
well another somewhat sleepless night but...
I have gotten into a little bit of a "talk" with skiplink via text message to his cell phone.
and I have talked to a couple of people at TelX (they have been very helpful).
as it stands here is my options:
Obviously option #2 is the best, and I'm convincing skiplink to allow me to do this (e.g. give me something identifiable about the servers). #3 is another, albeit expensive option, however would put me at risk of potentially damaging other peoples servers and being held liable. and #4 would be my fall back option of last resort. these seem to be my only options short of hiring a lawyer to help with #2.
I'd like to talk the staff at TelX for being very helpful and professional about the whole thing. Hopefully Skiplink will get back to me with some serial numbers soon. once again i'll keep you all posted; and thanks again for everyones support.
P.S. a couple people have suggested a 5th option of hiring Ninjas and/or Pirates to retrieve the data... I'll work on that.'
Confirmed that skiplink is locked out due to lack of payment. Talked to a few people at TelX, best they can do for me at this point is power up the entire cage (I pay for it) and hope my stuff turns on etc. however I also got a few text messages from Skiplink and he has obviously already canceled the network connections into the unit and there would be no chance of them being visible. so I'll keep trying to convince skiplink to tell me the serial numbers from the boxes or something else to identify them so i can actually get the boxes hooked up. *sigh*
I'll keep everyone posted if I get anywhere.
Well I've been reading through my backlog of about 200 emails, and i found one from someone who has said that skiplinks assets are being held by TelX for non-payment of bills. if this is not resolved I might be SOL. If anyone happens to know anyone who works in that building that might be able to "borrow" the server or the hard drives for a few hours so that I might be able to just copy off the data it would be greatly appreciated.
Still nothing from skiplink, i REALLY wish they would just get back to me with something, or a timeframe for getting back up.
Thanks for everyone who is offering hosting, I have figured out what I'm going to do once i can get ahold of the data again. for now there is nothing to do but wait.
No word from skiplink today, They are being very unprofessional and this is pissing me off. I will be moving the servers ASAP once i get them back up at skiplink so that i can move the date.
Right now it looks like i'm going to move to Amazon EC2. this looks to be the best solution and should in theory actually reduce my costs monthly by a couple hundred dollars. and if i can later set it up to start and kill servers at peak times i should be able to save a little more again.
All this aside I have definitely learned the use of off-site backups. once the servers are moved I will be adding in some backup scripts to send the backups around the world. I'll have some stats of what I would need for backups later and ask around to see if anyone would be willing to host backups.
I got a few text messages from the owner FINALLY.. seems the hardware failure for the building was unrelated. skiplink apparently had a disagreement with someone over something and therefor my boxes are offline. they will be moving the boxes to a new location hopefully tomorrow. needless to say I am not impressed, and I will be moving to a new host ASAP. (right now I'm thinking about EC2)
No news yet, still no word from skiplink, and only some 2nd hand knowledge from inside the building. I have to head out for the night, if anyone has some important info, email me i'll have my phone with me.'
Update: well a guy who knows a guy has informed me that the entire building is down. (thanks Jeff) No real specifics on why, but everything is down and I'm waiting to hear back with more information. at least i can be relatively sure the data is safe, just unaccessible.
when the server went down at first I dident think anything was wrong with the site. I attempted to call Skiplink (my server provider) to find out if anything was wrong. They dident answer, but it was 1 AM, so i left it for the time being.
The next day In the morning the server still was not up, so I gave them another call, they did not answer. It was at this point I noticed that my 2nd server (same company, different carrier, different physical rack) was also not up. And I this is where i start panicking... the 2nd server was the backups for the first.
I have emailed, txt'd, phoned (5 different numbers) and I have yet to get ahold of anyone. If anyone knows anyone that works at http://www.56marietta.com, this is the building where the servers are housed.. could you get them to find out if anything is going on there and shoot me an email. I would REALLY appreciate it.
I'll go freak out on my own now and update this page whenever I find out anything new.
P.S. come back for updates if i have any.
Mike Valstar mikevalstar@gmail.com