Server troubles for indymedia

Okay. This sounds like typical IT dept apologies for universal IT problems and it sounds like you have dealt with it in a fairly professional manner. All IT depts are understaffed and most of them are underfunded, services do go down from time to time, no need to panic too much.

Even your average corporate IT dept has a wealth of free, external support and resources. Indymedia should be able to draw on extra support simply due to it's good reputation. For those of you that haven't been suits I should stress a surprising number of suits do read Indymedia - at least on their home PCs where they feel safe, the Uni students who know even more and have more spare time for the most part consider themselves progressive and all open-source folk work for egoboo (namechecking them is enough).

Training non-techs into techs is too time consuming while there are so many sympathetic techs out there that you haven't contacted. Sure, workshops such as Activix are essential for the individual activists but secondary imho to IMs needs, retraining existing techs is far quicker. I would suggest an active recruitment campaign, not just this thread but in university and college computer depts. These folks have to think up projects anyway to pass their qualifications and need the credibility to get laid. Someone come up with a poster, a standard request for help, so that people here in college and uni can print it off and post it on virtual and actual uni/college noticeboards. Something along the lines of 'Help IM and you'll get egoboo, overthrow the government and meet girls' should work.

Only a major and unresolvable crisis should be posted on the IM newswire - most readers here aren't interested and you can't blame them. I imagine you have your own backroom forums and newsletters and such. To successfully integrate part-time volunteers you should be posting those details here and now. I've resurrected a few dead disks and RAID systems in my time but I couldn't know about this particular issue until after the event. If you have a few thousand folk like me then only one or two of us need to be available at any given time. I would suggest a techie-mailing list and a forum, one for code development, one for code support and one for hardware support. I suggest your existing techies do this or else newbies like me will be ignorant of existing procedures. It is often easier to start from scratch, not that I'm recommending that.

For any interactive system the size of UK IM I would suggest cheap, refurbished, clustered PCs. The most powerful supercomputer ever was just a bunch of crappy PCs clustered together. As soon as any individual PC bursts into flames, you just disconnect it and plug in a newly imaged replacement. With standard back-ups and PCs in every council dump in the UK it eludes me why you have lost your wiki document server.
1) Identify and replace the damaged hardware component. - 2 hours
2) Reimage the system - 30 minutes
3) Restore last nights backup - 30 minutes

Since that hasn't happened I'm guessing the problems are more fundamental than you realise.

Anyway, my main point is there is already a wealth of voluntary help out there that other groups regularly mine. It is difficult for any group to cope with a poorly designed or obscure system. Instead of following US IM tech trends you need to go to the source - Sourceforge has everything I'd need if I wanted to recreate your entire site from scratch in 6 months, including third-line/developer support, and my local dump has all the hardware I'd require. Think ease-of maintenance, think ongoing external support, think disposable hardware.

Being more adventurous, and if there is someone cleverer than me doing it, you could gain extra computing power using the Seti-screensaver model.
With the amount of individuals and groups who are already producing high-quality video content, with P2P and resource sharing, cheap technology going rusty and with the amount of political dissent in the UK I see no real reason why Indymedia could not be competing with Channel 4/BBC 2 for mainstream audiences in two years. Text is fine for the converted but the lumpen bourgeousie require visual programming (excuse the pun). Once you have that under your belt, I suggest you start building your own ISP - free internet access for any user who helps along the existing and growing community WLAN model.

for some time there have been weekly trainings in the rampart hacklab. due to lack of attendance they will be monthly rather than weekly, and always designed for the not-so-techies.

Next one will be Sunday 29th... best might be to email the address on this post to see what is really on offer. there is a link to these workshops at the top right corner of the london indymedia page anyway (open ofor everyone)

 http://wiki.hacklab.org.uk/index.php/Workshops/OpentoEveryone

Your comment is totally demeaning to the incredible skills that most IMC techs have. If there's many others voicing ignorant views like you then if I were an overworked IMC tech I'd consider leaving. People with an arty background or no professional background just have NO idea at all of the depth of skills needed to build and maintain a complex network like IMC. Techs could earn big money and work long hours using such skills in mainstream industry. Your belief that these skills can be passed on with a few pisspot 2 hour workshops is laughable and downright insulting to anyone who's taken the trouble to acquire these skills with YEARS of hard graft. You need to realise that good techs are in a completely different league to normal people as regards intelligence and knowledge. This is one thing that's really annoying about activism today - so many politicos with an attention span measured in minutes think ANY skill can be passed on in a few hours of waffly, touchy-feely, fluffy workshops. Dream on.

So let's see now how long to really learn all this stuff:
Thorough computer networking skills TCP/IP and all the other mind boggling protocols - several months
Thorough knowledge of Linux internals, command line, kernel stuff etc - several months
Thorough knowledge of good general purpose programming techniques- about a year
Thorough knowledge of HTML, XML and web design - several months
Thorough knowledge of Java, SQL, PHP and whatever other web languages - several months
Understanding how the hell Mir works - who knows?

Quite a chunk of anyone's life isn't it? Do you think it'd be ok for existing techs to take that sort of time off to teach a bunch of newbies those skills over that time period? Hardly which is why, regrettably, there are no workshops and complete newbies need not apply. The only realistic chance for change is for people who already have considerable skills to get involved. The only way to attract and keep such people onboard is for EVERYONE to show all techs far more respect, constant gratitude and recognition than is shown to them at present. I'm frankly amazed at how techs are pretty much completely taken for granted with barely murmured gratitude, relegated to the background, and often treated with suspicion presumably because they are seen to have so much potential control of the network - not that I've ever seen any of them abuse that control.

The reason why you probably receive a "muted response" from techs is because they know damn well you just wouldn't understand fully what they'd need to tell you and they don't have sufficient time in their hopelessly overworked lives to translate the stuff into sufficiently simple terms for you to understand. Simple innit? No, there are few simple things in this technological world.

So remember, if we want to solve this log jam, all of us need to start being incredibly nice to existing and any would be techs.
So let's hear it for them - R E S P E C T!

> Only a major and unresolvable crisis should be posted on the IM newswire - most readers here aren't
> interested and you can't blame them.

That may be true, but how are those that might be interested and able to help going to find out about problems if they are not publicly announced?

> I would suggest a techie-mailing list and a forum, one for code development, one for code support
> and one for hardware support.

Indymedia has those lists already, and many many more but there isn't much point in sending people to these lists if they haven't first got in contact with they local collective. Obviously indymedia isn't going to recruit complete strangers online and had over root access on an email list ;-)

> For any interactive system the size of UK IM I would suggest cheap, refurbished, clustered PCs.

Web servers don't need to be super computers and clusters are much use when you pay for rack space in a commercial web hosting service. What is needed is large amounts of RAM and you don't find that in the council dump.

> With standard back-ups and PCs in every council dump in the UK it eludes me why you have lost
> your wiki document server.

Because, like any complex system is prone to do eventually, it broke! You might think that it could be quickly repaired but first somebody had to notice the problem and report it to people that might know something about it. Bearing in mind the global nature of indymedia and the time zones involved, it could be at least 12 hours before somebody in a position to do something actually gets to hear about it. People need to establish who has root access, who has physical access, who has the diagnosis skills required, who has transport or time to go to where the hardware is etc etc.

> Since that hasn't happened I'm guessing the problems are more fundamental than you realise.

Nope, the problem is obvious. There are not enough techies actively involved in Indymedia to ensure that such problem can be quickly and efficiently dealt with.

> It is difficult for any group to cope with a poorly designed or obscure system. Instead of following
> US IM tech trends you need to go to the source - Sourceforge has everything I'd need if I wanted to
> recreate your entire site from scratch in 6 months, including third-line/developer support.

There are only a handful of open source content management systems suitable for running an open publishing systems like the 200 or so indymedia sites around the world. The one used by the UK is written from scratch and designed to address key requirement of the growing indymedia network. It is the only one of the IMC codebases that provides undemanding mirroring. It is not that the UK is blindly following US trends - in fact you might be interested to learn that both the global site and New York IMC among others are actually hosted in the UK on the same server as the IMC UK site.

> With the amount of individuals and groups who are already producing high-quality video content,
> with P2P and resource sharing, cheap technology going rusty and with the amount of political dissent
> in the UK I see no real reason why Indymedia could not be competing with Channel 4/BBC 2
> for mainstream audiences in two years.

Well, I can think of a quite a few reasons. Lets learn to crawl first hey, before we try making a dash for the finishing line.

> Text is fine for the converted but the lumpen bourgeousie require visual programming (excuse the pun).
> Once you have that under your belt, I suggest you start building your own ISP - free internet access
> for any user who helps along the existing and growing community WLAN model.

Yes, well, fine ambitions no doubt, but lets see if we can sort out a few fundamental things first - like a working search engine, faster publish server, a radio server that stays up for more than a week at a time etc.

Non Tech IMCists says that during the last couple of years of being active with indymedia they have never come across a workshop or tutorial about technical issues. Well, I have. I remember training about CVS and templates at a gathering in Bristol during the MIR migration (ironic because Bristol IMC never did join the MIR network).

Non Tech says, "whenever i or other imcistas ask about technical stuff there's often a muted response that mostly includes some technical shorthand that we then have to spend hours looking up!". Well, there is a reason for shorthand, it sums up knowledge that takes years to acquire. As well as underestimating the vast amount of time it takes to establish such knowledge, Non Tech also underestimates the skill required to be a good teacher. Techies might have lots of tech stills abd be willing to share what they have learnt, but that doesn't mean that they are good teachers and who is going to teach them how to teach?

Unlike 'someone with more of a clue', I don't think we can solve this log jam simply by "being incredibly nice to existing and any would be techs". I also think that they are respected by most people actively involved (although clearly too much is expected of them). What is needed is more people in the collectives so that the limited number of techies can concentrate on the techie things while others take care of the things that they can do. I don't suggest strengthening the techie divide, although I can see that what I have said does sound like it. I think that if the strain was taken off the techies then more people would become trained up at some of the more achievable techie tasks and thus start to close the gap. It is not realistic to turn not techies into programmers in a few workshops but it is possible for techies to be freed from creating template, or doing admin, or moderating mailing lists etc by people who will start to develop techie skills by doing these tasks.

I'm not sure what's up with the other servers, but I can comment on web1 & web4 which are part of ahimsa*.

Poor little ahimsa-web1 is an 866MHz, 1 gig RAM computer serving a bunch of SPIP, sf-active, and custom dynamic sites.

Some changes were made to video.indymedia.org's code so when someone uploaded new files it just clobbered the processor. Once it was figured out what the culprit was, video.indy was moved to ahimsa-web4. Like someone above suggested--we have a "cluster" so we can move bits around where they are needed. I don't know where "web4 didn't have a filesystem" came from as it most certainly does. It serves up many sites including italy.indymedia.org. In the process of moving video.indy to the new box it also had it's files moved to the brand spankin' new fileserver which is /much/ faster than the old one (dual 3ware 9xxx controllers with 12x 400 gig drives in RAID10 for you geeks). Anyway, video.indy is back up, on a faster webserver, feeding off a faster fileserver with even more space. Good thing.

Back to ahimsa-web1: a few times I've sent emails saying "uh, we need to move these dynamic sites and stuff them into a vserver" but it hasn't happened. There is a bit of inertia ("it's working now") and timing it is always a pain. In the end, I'll probably just move them unilaterally...

We do need more geographic server diversity though. Set up a webserver and start mirroring Mir sites, would be a nice start. :)

-Jeff