The last two weeks have seen multiple server troubles for Indymedia, this time however it's not been the FBI trying to throw a spanner in the works...
About two weeks ago one of the indymedia servers which is based in London, died, and took with it the documentation wiki for the entire global indymedia network. No actually indymedia websites were effected but nether-the-less it caused great annoyance around the world to those that use it.
Then, less than a week later and with the documentation server still offline with a RAID failure, the server known as 'web1' starting falling. This server has been under great strain for some time and hosts quite a number of indymedia sites. IMC techies quickly got the server and the effected IMCs back online but the trouble continued. The video.indymedia.org site was identified as contributed the highest load on the server resouces and so it was shut down so that the other sites could remain online until and better solution was found.
It was decided to move ahead and move the IVDN video site to another higher spec server, which has the highly boring name 'web4' but this server was not yet ready to be commissioned and didn't even contain a file system.
They say trouble comes in threes and it certainly did for the indymedia servers this week. With docs and video still offline, radio.indymedia.org vanished as well. The radio site is hosted on a server called berkman, the third indymedia service to go offline during the last two weeks. Of the three it was the quickest to be resolved and is now back online and serving dozens of indymedia radio streams from around the globe.
Meanwhile the news appears to be that docs should be up soon after being repaired by London techs and the video site is being migrated to it's new host and should hopefully be online by the time you read this.
These incidents highlight one thing, and no, this reports isn't going to end with a plea for money to buy new servers. The problem is lack of techies with time to and skills to contribute to keeping indymedia servers running with a minimal of downtime. The servers are not necessarily located where indymedia techies are instantly available so delays become unavoidable when trying to deal with repairs. Additionally, techies are over-burdened with demands from many IMCs and unrelated projects, as well as having commitments to work and family.
The bottom line is that indymedia needs more techies, so if you think you could help please email imc-uk-tech (if you are in the UK) or imc-tech (if you are somewhere else).
Then, less than a week later and with the documentation server still offline with a RAID failure, the server known as 'web1' starting falling. This server has been under great strain for some time and hosts quite a number of indymedia sites. IMC techies quickly got the server and the effected IMCs back online but the trouble continued. The video.indymedia.org site was identified as contributed the highest load on the server resouces and so it was shut down so that the other sites could remain online until and better solution was found.
It was decided to move ahead and move the IVDN video site to another higher spec server, which has the highly boring name 'web4' but this server was not yet ready to be commissioned and didn't even contain a file system.
They say trouble comes in threes and it certainly did for the indymedia servers this week. With docs and video still offline, radio.indymedia.org vanished as well. The radio site is hosted on a server called berkman, the third indymedia service to go offline during the last two weeks. Of the three it was the quickest to be resolved and is now back online and serving dozens of indymedia radio streams from around the globe.
Meanwhile the news appears to be that docs should be up soon after being repaired by London techs and the video site is being migrated to it's new host and should hopefully be online by the time you read this.
These incidents highlight one thing, and no, this reports isn't going to end with a plea for money to buy new servers. The problem is lack of techies with time to and skills to contribute to keeping indymedia servers running with a minimal of downtime. The servers are not necessarily located where indymedia techies are instantly available so delays become unavoidable when trying to deal with repairs. Additionally, techies are over-burdened with demands from many IMCs and unrelated projects, as well as having commitments to work and family.
The bottom line is that indymedia needs more techies, so if you think you could help please email imc-uk-tech (if you are in the UK) or imc-tech (if you are somewhere else).
Comments
Hide the following 6 comments
a hard drive
20.03.2006 23:19
Even your average corporate IT dept has a wealth of free, external support and resources. Indymedia should be able to draw on extra support simply due to it's good reputation. For those of you that haven't been suits I should stress a surprising number of suits do read Indymedia - at least on their home PCs where they feel safe, the Uni students who know even more and have more spare time for the most part consider themselves progressive and all open-source folk work for egoboo (namechecking them is enough).
Training non-techs into techs is too time consuming while there are so many sympathetic techs out there that you haven't contacted. Sure, workshops such as Activix are essential for the individual activists but secondary imho to IMs needs, retraining existing techs is far quicker. I would suggest an active recruitment campaign, not just this thread but in university and college computer depts. These folks have to think up projects anyway to pass their qualifications and need the credibility to get laid. Someone come up with a poster, a standard request for help, so that people here in college and uni can print it off and post it on virtual and actual uni/college noticeboards. Something along the lines of 'Help IM and you'll get egoboo, overthrow the government and meet girls' should work.
Only a major and unresolvable crisis should be posted on the IM newswire - most readers here aren't interested and you can't blame them. I imagine you have your own backroom forums and newsletters and such. To successfully integrate part-time volunteers you should be posting those details here and now. I've resurrected a few dead disks and RAID systems in my time but I couldn't know about this particular issue until after the event. If you have a few thousand folk like me then only one or two of us need to be available at any given time. I would suggest a techie-mailing list and a forum, one for code development, one for code support and one for hardware support. I suggest your existing techies do this or else newbies like me will be ignorant of existing procedures. It is often easier to start from scratch, not that I'm recommending that.
For any interactive system the size of UK IM I would suggest cheap, refurbished, clustered PCs. The most powerful supercomputer ever was just a bunch of crappy PCs clustered together. As soon as any individual PC bursts into flames, you just disconnect it and plug in a newly imaged replacement. With standard back-ups and PCs in every council dump in the UK it eludes me why you have lost your wiki document server.
1) Identify and replace the damaged hardware component. - 2 hours
2) Reimage the system - 30 minutes
3) Restore last nights backup - 30 minutes
Since that hasn't happened I'm guessing the problems are more fundamental than you realise.
Anyway, my main point is there is already a wealth of voluntary help out there that other groups regularly mine. It is difficult for any group to cope with a poorly designed or obscure system. Instead of following US IM tech trends you need to go to the source - Sourceforge has everything I'd need if I wanted to recreate your entire site from scratch in 6 months, including third-line/developer support, and my local dump has all the hardware I'd require. Think ease-of maintenance, think ongoing external support, think disposable hardware.
Being more adventurous, and if there is someone cleverer than me doing it, you could gain extra computing power using the Seti-screensaver model.
With the amount of individuals and groups who are already producing high-quality video content, with P2P and resource sharing, cheap technology going rusty and with the amount of political dissent in the UK I see no real reason why Indymedia could not be competing with Channel 4/BBC 2 for mainstream audiences in two years. Text is fine for the converted but the lumpen bourgeousie require visual programming (excuse the pun). Once you have that under your belt, I suggest you start building your own ISP - free internet access for any user who helps along the existing and growing community WLAN model.
tech
there are trainings in rampart
21.03.2006 15:19
Next one will be Sunday 29th... best might be to email the address on this post to see what is really on offer. there is a link to these workshops at the top right corner of the london indymedia page anyway (open ofor everyone)
rampart hacklab
e-mail:
londontrainings@riseup.net
Homepage:
http://rampart.org.uk
non tech imcista has bad attitude and no clue
25.03.2006 02:01
So let's see now how long to really learn all this stuff:
Thorough computer networking skills TCP/IP and all the other mind boggling protocols - several months
Thorough knowledge of Linux internals, command line, kernel stuff etc - several months
Thorough knowledge of good general purpose programming techniques- about a year
Thorough knowledge of HTML, XML and web design - several months
Thorough knowledge of Java, SQL, PHP and whatever other web languages - several months
Understanding how the hell Mir works - who knows?
Quite a chunk of anyone's life isn't it? Do you think it'd be ok for existing techs to take that sort of time off to teach a bunch of newbies those skills over that time period? Hardly which is why, regrettably, there are no workshops and complete newbies need not apply. The only realistic chance for change is for people who already have considerable skills to get involved. The only way to attract and keep such people onboard is for EVERYONE to show all techs far more respect, constant gratitude and recognition than is shown to them at present. I'm frankly amazed at how techs are pretty much completely taken for granted with barely murmured gratitude, relegated to the background, and often treated with suspicion presumably because they are seen to have so much potential control of the network - not that I've ever seen any of them abuse that control.
The reason why you probably receive a "muted response" from techs is because they know damn well you just wouldn't understand fully what they'd need to tell you and they don't have sufficient time in their hopelessly overworked lives to translate the stuff into sufficiently simple terms for you to understand. Simple innit? No, there are few simple things in this technological world.
So remember, if we want to solve this log jam, all of us need to start being incredibly nice to existing and any would be techs.
So let's hear it for them - R E S P E C T!
someone with more of a clue
Fine ambitions?
25.03.2006 02:44
> interested and you can't blame them.
That may be true, but how are those that might be interested and able to help going to find out about problems if they are not publicly announced?
> I would suggest a techie-mailing list and a forum, one for code development, one for code support
> and one for hardware support.
Indymedia has those lists already, and many many more but there isn't much point in sending people to these lists if they haven't first got in contact with they local collective. Obviously indymedia isn't going to recruit complete strangers online and had over root access on an email list ;-)
> For any interactive system the size of UK IM I would suggest cheap, refurbished, clustered PCs.
Web servers don't need to be super computers and clusters are much use when you pay for rack space in a commercial web hosting service. What is needed is large amounts of RAM and you don't find that in the council dump.
> With standard back-ups and PCs in every council dump in the UK it eludes me why you have lost
> your wiki document server.
Because, like any complex system is prone to do eventually, it broke! You might think that it could be quickly repaired but first somebody had to notice the problem and report it to people that might know something about it. Bearing in mind the global nature of indymedia and the time zones involved, it could be at least 12 hours before somebody in a position to do something actually gets to hear about it. People need to establish who has root access, who has physical access, who has the diagnosis skills required, who has transport or time to go to where the hardware is etc etc.
> Since that hasn't happened I'm guessing the problems are more fundamental than you realise.
Nope, the problem is obvious. There are not enough techies actively involved in Indymedia to ensure that such problem can be quickly and efficiently dealt with.
> It is difficult for any group to cope with a poorly designed or obscure system. Instead of following
> US IM tech trends you need to go to the source - Sourceforge has everything I'd need if I wanted to
> recreate your entire site from scratch in 6 months, including third-line/developer support.
There are only a handful of open source content management systems suitable for running an open publishing systems like the 200 or so indymedia sites around the world. The one used by the UK is written from scratch and designed to address key requirement of the growing indymedia network. It is the only one of the IMC codebases that provides undemanding mirroring. It is not that the UK is blindly following US trends - in fact you might be interested to learn that both the global site and New York IMC among others are actually hosted in the UK on the same server as the IMC UK site.
> With the amount of individuals and groups who are already producing high-quality video content,
> with P2P and resource sharing, cheap technology going rusty and with the amount of political dissent
> in the UK I see no real reason why Indymedia could not be competing with Channel 4/BBC 2
> for mainstream audiences in two years.
Well, I can think of a quite a few reasons. Lets learn to crawl first hey, before we try making a dash for the finishing line.
> Text is fine for the converted but the lumpen bourgeousie require visual programming (excuse the pun).
> Once you have that under your belt, I suggest you start building your own ISP - free internet access
> for any user who helps along the existing and growing community WLAN model.
Yes, well, fine ambitions no doubt, but lets see if we can sort out a few fundamental things first - like a working search engine, faster publish server, a radio server that stays up for more than a week at a time etc.
reality bytes
A more diplomatic clue
25.03.2006 03:26
Non Tech says, "whenever i or other imcistas ask about technical stuff there's often a muted response that mostly includes some technical shorthand that we then have to spend hours looking up!". Well, there is a reason for shorthand, it sums up knowledge that takes years to acquire. As well as underestimating the vast amount of time it takes to establish such knowledge, Non Tech also underestimates the skill required to be a good teacher. Techies might have lots of tech stills abd be willing to share what they have learnt, but that doesn't mean that they are good teachers and who is going to teach them how to teach?
Unlike 'someone with more of a clue', I don't think we can solve this log jam simply by "being incredibly nice to existing and any would be techs". I also think that they are respected by most people actively involved (although clearly too much is expected of them). What is needed is more people in the collectives so that the limited number of techies can concentrate on the techie things while others take care of the things that they can do. I don't suggest strengthening the techie divide, although I can see that what I have said does sound like it. I think that if the strain was taken off the techies then more people would become trained up at some of the more achievable techie tasks and thus start to close the gap. It is not realistic to turn not techies into programmers in a few workshops but it is possible for techies to be freed from creating template, or doing admin, or moderating mailing lists etc by people who will start to develop techie skills by doing these tasks.
respect all round
ahimsa-web1 & ahimsa-web4
28.03.2006 19:43
Poor little ahimsa-web1 is an 866MHz, 1 gig RAM computer serving a bunch of SPIP, sf-active, and custom dynamic sites.
Some changes were made to video.indymedia.org's code so when someone uploaded new files it just clobbered the processor. Once it was figured out what the culprit was, video.indy was moved to ahimsa-web4. Like someone above suggested--we have a "cluster" so we can move bits around where they are needed. I don't know where "web4 didn't have a filesystem" came from as it most certainly does. It serves up many sites including italy.indymedia.org. In the process of moving video.indy to the new box it also had it's files moved to the brand spankin' new fileserver which is /much/ faster than the old one (dual 3ware 9xxx controllers with 12x 400 gig drives in RAID10 for you geeks). Anyway, video.indy is back up, on a faster webserver, feeding off a faster fileserver with even more space. Good thing.
Back to ahimsa-web1: a few times I've sent emails saying "uh, we need to move these dynamic sites and stuff them into a vserver" but it hasn't happened. There is a bit of inertia ("it's working now") and timing it is always a pain. In the end, I'll probably just move them unilaterally...
We do need more geographic server diversity though. Set up a webserver and start mirroring Mir sites, would be a nice start. :)
-Jeff
jeff
e-mail:
jeff@indymedia.org
Homepage:
http://jebba.blagblagblag.org