Skip to content or view screen version

Server troubles for indymedia

cvs | 17.03.2006 04:40 | Indymedia | Technology | Zapatista | London | World

The last two weeks have seen multiple server troubles for Indymedia, this time however it's not been the FBI trying to throw a spanner in the works...

About two weeks ago one of the indymedia servers which is based in London, died, and took with it the documentation wiki for the entire global indymedia network. No actually indymedia websites were effected but nether-the-less it caused great annoyance around the world to those that use it.

Then, less than a week later and with the documentation server still offline with a RAID failure, the server known as 'web1' starting falling. This server has been under great strain for some time and hosts quite a number of indymedia sites. IMC techies quickly got the server and the effected IMCs back online but the trouble continued. The video.indymedia.org site was identified as contributed the highest load on the server resouces and so it was shut down so that the other sites could remain online until and better solution was found.

It was decided to move ahead and move the IVDN video site to another higher spec server, which has the highly boring name 'web4' but this server was not yet ready to be commissioned and didn't even contain a file system.

They say trouble comes in threes and it certainly did for the indymedia servers this week. With docs and video still offline, radio.indymedia.org vanished as well. The radio site is hosted on a server called berkman, the third indymedia service to go offline during the last two weeks. Of the three it was the quickest to be resolved and is now back online and serving dozens of indymedia radio streams from around the globe.

Meanwhile the news appears to be that docs should be up soon after being repaired by London techs and the video site is being migrated to it's new host and should hopefully be online by the time you read this.

These incidents highlight one thing, and no, this reports isn't going to end with a plea for money to buy new servers. The problem is lack of techies with time to and skills to contribute to keeping indymedia servers running with a minimal of downtime. The servers are not necessarily located where indymedia techies are instantly available so delays become unavoidable when trying to deal with repairs. Additionally, techies are over-burdened with demands from many IMCs and unrelated projects, as well as having commitments to work and family.

The bottom line is that indymedia needs more techies, so if you think you could help please email imc-uk-tech (if you are in the UK) or imc-tech (if you are somewhere else).



cvs

Additions

share the wealth then!

17.03.2006 10:57

We all appreciate the enormous commitment tech imcistas put into the imc project and it's obvious we need more technically orientated people but the tech imcistas also have a responsibility to share the wealth and empower and train more people to become technically proficient. You just can't assume people can do it all on their own - Do It Together rather than Do it Yourself should be an ethos we embrace.

For the last couple of years that i've been active as an imcista i have never really come across a workshop or tutorial about technical issues - please correct me if i'm wrong here. And whenever i or other imcistas ask about technical stuff there's often a muted response that mostly includes some technical shorthand that we then have to spend hours looking up!

If the rest of us non-technical imcistas responded like this to other areas of indymedia like film- making or writing reports then I'm sure no-one would feel empowered to contribute. For example, there are simple and complicated ways of explaining codecs or for that matter how to write a good article. Sorry if this offends anyone but that's how i've experienced it.

Perhaps the time is ripe to really reflect on how we share information technically as a network and think more inclusively about this? We can't just rely and assume that people will find out technical stuff all on their own and that's why i reckon we've got this skills shortage.

Simple really innit?

non tech imcista


Simple, innit?

17.03.2006 16:04

Through being involved with indymedia, I have learned: Using and admining mailing lists, basic html, using digital camera, basic video editing, contributing to the maintenance of an open publishing website, using chatrooms, using wikis, writing features, installing simple linux distributions, using openoffice, basic command line use, and probably some more things. Reading the f****** manual was the most important, and I would advise you to do the same.

There are many local imc groups, linux user groups, media groups. If you really want to learn, get in touch with them and you will be surprised how much there is to pick up.

florent


Not so simple

17.03.2006 16:35

Come now, be realistic, it's not as simple as that. Do you seriously think that a person with no background in programming and hardware diagnostics would pick up these skill by going to a couple of skill sharing workshops?

Programming is highly skilled profession which takes considerable time to master and those skills must be constantly maintained as they become outdated and redundant quickly. Usually, people that have these skills are using them professionally and not merely as a hobby. Programmers learn their skills over years spent teaching themselves, reading, practicing and evolving from one language to another. It's not like using a camera or video editing where a complete beginner can be taught the basics with a day and start producing their own material.

Likewise with computer hardware technicians. Most are self taught from building and maintaining their own machines or from employement. You could quickly learn some basic skills, enough to diagnose some simply problems but if you don't really have a technical background and intend to continue to exercise such skills on a regular basis then you will not really progress. Everything changes pretty quickly in computer hardware so you have to continue to use the skills or they become redundant.

That doesn't mean that there is no place for indymedia training for techincal skills and in fact there have been quite a few indymedia related training events for people to share tech skills. For example, the resent gatherings about video included sessions on encoding and uploading to indymedia, regular radio workshops in the radio studio at rampART taught people not only how to record and mix audio but also how to compress and upload. However, these skills are fundamentally different from the kind of advanced technical skills I think we are talking about here.

Nether-the-less, there are a number of hacklabs around the country were people could go to share and develop such skills if they wished but I don't see much of that happening or even much demand for it unfortunately.

Additionally there are the indymedia IRC channels were people regularly request and receive help on technical issues and share their skills or diagnose problems. Clearly there is a willingness by techies to share their skills and it is in their interest to do so in terms of indymedia work since it is not a competition over contracts and wages.

p.s. There are a couple more articles on these issues in the newswire at the moment...
see

Indymedia needs you
 https://publish.indymedia.org.uk/en/2006/03/336029.html

and
hacklab report
 https://publish.indymedia.org.uk/en/2006/03/335993.html










ben


Manuals

17.03.2006 20:30

There is a _lot_ to learn, you can do it a page at a time, It helps to know where to look..
I have found these resources useful;

1, a good intro linux  http://www.openicdl.org.za/courseware.html#releasecourseware

2, a intro to the command line  http://www.linuxcommand.org/

3, This is pretty much everything you need to know
 http://rute.2038bug.com/index.html.gz

4, This covers webservers see in particular section 36 Apache servers
 http://rute.2038bug.com/node39.html.gz

geek


Comments

Hide the following 6 comments

a hard drive

20.03.2006 23:19

Okay. This sounds like typical IT dept apologies for universal IT problems and it sounds like you have dealt with it in a fairly professional manner. All IT depts are understaffed and most of them are underfunded, services do go down from time to time, no need to panic too much.

Even your average corporate IT dept has a wealth of free, external support and resources. Indymedia should be able to draw on extra support simply due to it's good reputation. For those of you that haven't been suits I should stress a surprising number of suits do read Indymedia - at least on their home PCs where they feel safe, the Uni students who know even more and have more spare time for the most part consider themselves progressive and all open-source folk work for egoboo (namechecking them is enough).

Training non-techs into techs is too time consuming while there are so many sympathetic techs out there that you haven't contacted. Sure, workshops such as Activix are essential for the individual activists but secondary imho to IMs needs, retraining existing techs is far quicker. I would suggest an active recruitment campaign, not just this thread but in university and college computer depts. These folks have to think up projects anyway to pass their qualifications and need the credibility to get laid. Someone come up with a poster, a standard request for help, so that people here in college and uni can print it off and post it on virtual and actual uni/college noticeboards. Something along the lines of 'Help IM and you'll get egoboo, overthrow the government and meet girls' should work.

Only a major and unresolvable crisis should be posted on the IM newswire - most readers here aren't interested and you can't blame them. I imagine you have your own backroom forums and newsletters and such. To successfully integrate part-time volunteers you should be posting those details here and now. I've resurrected a few dead disks and RAID systems in my time but I couldn't know about this particular issue until after the event. If you have a few thousand folk like me then only one or two of us need to be available at any given time. I would suggest a techie-mailing list and a forum, one for code development, one for code support and one for hardware support. I suggest your existing techies do this or else newbies like me will be ignorant of existing procedures. It is often easier to start from scratch, not that I'm recommending that.

For any interactive system the size of UK IM I would suggest cheap, refurbished, clustered PCs. The most powerful supercomputer ever was just a bunch of crappy PCs clustered together. As soon as any individual PC bursts into flames, you just disconnect it and plug in a newly imaged replacement. With standard back-ups and PCs in every council dump in the UK it eludes me why you have lost your wiki document server.
1) Identify and replace the damaged hardware component. - 2 hours
2) Reimage the system - 30 minutes
3) Restore last nights backup - 30 minutes

Since that hasn't happened I'm guessing the problems are more fundamental than you realise.

Anyway, my main point is there is already a wealth of voluntary help out there that other groups regularly mine. It is difficult for any group to cope with a poorly designed or obscure system. Instead of following US IM tech trends you need to go to the source - Sourceforge has everything I'd need if I wanted to recreate your entire site from scratch in 6 months, including third-line/developer support, and my local dump has all the hardware I'd require. Think ease-of maintenance, think ongoing external support, think disposable hardware.

Being more adventurous, and if there is someone cleverer than me doing it, you could gain extra computing power using the Seti-screensaver model.
With the amount of individuals and groups who are already producing high-quality video content, with P2P and resource sharing, cheap technology going rusty and with the amount of political dissent in the UK I see no real reason why Indymedia could not be competing with Channel 4/BBC 2 for mainstream audiences in two years. Text is fine for the converted but the lumpen bourgeousie require visual programming (excuse the pun). Once you have that under your belt, I suggest you start building your own ISP - free internet access for any user who helps along the existing and growing community WLAN model.

tech


there are trainings in rampart

21.03.2006 15:19

for some time there have been weekly trainings in the rampart hacklab. due to lack of attendance they will be monthly rather than weekly, and always designed for the not-so-techies.

Next one will be Sunday 29th... best might be to email the address on this post to see what is really on offer. there is a link to these workshops at the top right corner of the london indymedia page anyway (open ofor everyone)

 http://wiki.hacklab.org.uk/index.php/Workshops/OpentoEveryone

rampart hacklab
mail e-mail: londontrainings@riseup.net
- Homepage: http://rampart.org.uk


non tech imcista has bad attitude and no clue

25.03.2006 02:01

Your comment is totally demeaning to the incredible skills that most IMC techs have. If there's many others voicing ignorant views like you then if I were an overworked IMC tech I'd consider leaving. People with an arty background or no professional background just have NO idea at all of the depth of skills needed to build and maintain a complex network like IMC. Techs could earn big money and work long hours using such skills in mainstream industry. Your belief that these skills can be passed on with a few pisspot 2 hour workshops is laughable and downright insulting to anyone who's taken the trouble to acquire these skills with YEARS of hard graft. You need to realise that good techs are in a completely different league to normal people as regards intelligence and knowledge. This is one thing that's really annoying about activism today - so many politicos with an attention span measured in minutes think ANY skill can be passed on in a few hours of waffly, touchy-feely, fluffy workshops. Dream on.

So let's see now how long to really learn all this stuff:
Thorough computer networking skills TCP/IP and all the other mind boggling protocols - several months
Thorough knowledge of Linux internals, command line, kernel stuff etc - several months
Thorough knowledge of good general purpose programming techniques- about a year
Thorough knowledge of HTML, XML and web design - several months
Thorough knowledge of Java, SQL, PHP and whatever other web languages - several months
Understanding how the hell Mir works - who knows?

Quite a chunk of anyone's life isn't it? Do you think it'd be ok for existing techs to take that sort of time off to teach a bunch of newbies those skills over that time period? Hardly which is why, regrettably, there are no workshops and complete newbies need not apply. The only realistic chance for change is for people who already have considerable skills to get involved. The only way to attract and keep such people onboard is for EVERYONE to show all techs far more respect, constant gratitude and recognition than is shown to them at present. I'm frankly amazed at how techs are pretty much completely taken for granted with barely murmured gratitude, relegated to the background, and often treated with suspicion presumably because they are seen to have so much potential control of the network - not that I've ever seen any of them abuse that control.

The reason why you probably receive a "muted response" from techs is because they know damn well you just wouldn't understand fully what they'd need to tell you and they don't have sufficient time in their hopelessly overworked lives to translate the stuff into sufficiently simple terms for you to understand. Simple innit? No, there are few simple things in this technological world.

So remember, if we want to solve this log jam, all of us need to start being incredibly nice to existing and any would be techs.
So let's hear it for them - R E S P E C T!

someone with more of a clue


Fine ambitions?

25.03.2006 02:44

> Only a major and unresolvable crisis should be posted on the IM newswire - most readers here aren't
> interested and you can't blame them.

That may be true, but how are those that might be interested and able to help going to find out about problems if they are not publicly announced?

> I would suggest a techie-mailing list and a forum, one for code development, one for code support
> and one for hardware support.

Indymedia has those lists already, and many many more but there isn't much point in sending people to these lists if they haven't first got in contact with they local collective. Obviously indymedia isn't going to recruit complete strangers online and had over root access on an email list ;-)

> For any interactive system the size of UK IM I would suggest cheap, refurbished, clustered PCs.

Web servers don't need to be super computers and clusters are much use when you pay for rack space in a commercial web hosting service. What is needed is large amounts of RAM and you don't find that in the council dump.

> With standard back-ups and PCs in every council dump in the UK it eludes me why you have lost
> your wiki document server.

Because, like any complex system is prone to do eventually, it broke! You might think that it could be quickly repaired but first somebody had to notice the problem and report it to people that might know something about it. Bearing in mind the global nature of indymedia and the time zones involved, it could be at least 12 hours before somebody in a position to do something actually gets to hear about it. People need to establish who has root access, who has physical access, who has the diagnosis skills required, who has transport or time to go to where the hardware is etc etc.

> Since that hasn't happened I'm guessing the problems are more fundamental than you realise.

Nope, the problem is obvious. There are not enough techies actively involved in Indymedia to ensure that such problem can be quickly and efficiently dealt with.

> It is difficult for any group to cope with a poorly designed or obscure system. Instead of following
> US IM tech trends you need to go to the source - Sourceforge has everything I'd need if I wanted to
> recreate your entire site from scratch in 6 months, including third-line/developer support.

There are only a handful of open source content management systems suitable for running an open publishing systems like the 200 or so indymedia sites around the world. The one used by the UK is written from scratch and designed to address key requirement of the growing indymedia network. It is the only one of the IMC codebases that provides undemanding mirroring. It is not that the UK is blindly following US trends - in fact you might be interested to learn that both the global site and New York IMC among others are actually hosted in the UK on the same server as the IMC UK site.

> With the amount of individuals and groups who are already producing high-quality video content,
> with P2P and resource sharing, cheap technology going rusty and with the amount of political dissent
> in the UK I see no real reason why Indymedia could not be competing with Channel 4/BBC 2
> for mainstream audiences in two years.

Well, I can think of a quite a few reasons. Lets learn to crawl first hey, before we try making a dash for the finishing line.

> Text is fine for the converted but the lumpen bourgeousie require visual programming (excuse the pun).
> Once you have that under your belt, I suggest you start building your own ISP - free internet access
> for any user who helps along the existing and growing community WLAN model.

Yes, well, fine ambitions no doubt, but lets see if we can sort out a few fundamental things first - like a working search engine, faster publish server, a radio server that stays up for more than a week at a time etc.

reality bytes


A more diplomatic clue

25.03.2006 03:26

Non Tech IMCists says that during the last couple of years of being active with indymedia they have never come across a workshop or tutorial about technical issues. Well, I have. I remember training about CVS and templates at a gathering in Bristol during the MIR migration (ironic because Bristol IMC never did join the MIR network).

Non Tech says, "whenever i or other imcistas ask about technical stuff there's often a muted response that mostly includes some technical shorthand that we then have to spend hours looking up!". Well, there is a reason for shorthand, it sums up knowledge that takes years to acquire. As well as underestimating the vast amount of time it takes to establish such knowledge, Non Tech also underestimates the skill required to be a good teacher. Techies might have lots of tech stills abd be willing to share what they have learnt, but that doesn't mean that they are good teachers and who is going to teach them how to teach?

Unlike 'someone with more of a clue', I don't think we can solve this log jam simply by "being incredibly nice to existing and any would be techs". I also think that they are respected by most people actively involved (although clearly too much is expected of them). What is needed is more people in the collectives so that the limited number of techies can concentrate on the techie things while others take care of the things that they can do. I don't suggest strengthening the techie divide, although I can see that what I have said does sound like it. I think that if the strain was taken off the techies then more people would become trained up at some of the more achievable techie tasks and thus start to close the gap. It is not realistic to turn not techies into programmers in a few workshops but it is possible for techies to be freed from creating template, or doing admin, or moderating mailing lists etc by people who will start to develop techie skills by doing these tasks.

respect all round


ahimsa-web1 & ahimsa-web4

28.03.2006 19:43

I'm not sure what's up with the other servers, but I can comment on web1 & web4 which are part of ahimsa*.

Poor little ahimsa-web1 is an 866MHz, 1 gig RAM computer serving a bunch of SPIP, sf-active, and custom dynamic sites.

Some changes were made to video.indymedia.org's code so when someone uploaded new files it just clobbered the processor. Once it was figured out what the culprit was, video.indy was moved to ahimsa-web4. Like someone above suggested--we have a "cluster" so we can move bits around where they are needed. I don't know where "web4 didn't have a filesystem" came from as it most certainly does. It serves up many sites including italy.indymedia.org. In the process of moving video.indy to the new box it also had it's files moved to the brand spankin' new fileserver which is /much/ faster than the old one (dual 3ware 9xxx controllers with 12x 400 gig drives in RAID10 for you geeks). Anyway, video.indy is back up, on a faster webserver, feeding off a faster fileserver with even more space. Good thing.

Back to ahimsa-web1: a few times I've sent emails saying "uh, we need to move these dynamic sites and stuff them into a vserver" but it hasn't happened. There is a bit of inertia ("it's working now") and timing it is always a pain. In the end, I'll probably just move them unilaterally...

We do need more geographic server diversity though. Set up a webserver and start mirroring Mir sites, would be a nice start. :)

-Jeff

jeff
mail e-mail: jeff@indymedia.org
- Homepage: http://jebba.blagblagblag.org