Fans fret as Fur Affinity nears fifth day of read-only mode
Nearly five days have passed since Fur Affinity entered read-only mode, and despite ongoing staff efforts, the interruption has led many users to mull alternatives.
The unheralded change, initiated in an attempt to resolve slowdowns during peak hours, caused consternation among artists, many of whom rely on the site for their income:
The problem is, this was unplanned downtime. […] If it was planned […] people would have had time to get things together. Such as commissioners, contests, bids, etc. and where to get in contact or if the bids/auctions are postponed while the site is down.
[…] at first I was worried that I wouldn't be able to make any money for christmas, but now I'm starting to worry that FA will be down so long that I won't be able to make enough for rent. […] it's so very disheartening when it was supposed to be a nice holiday season...
Operations team lead yak began an attempt to clear backlogged database transactions on Monday afternoon "as fast as the RAID10 array of 15k drives allow". A day later, FA status forum poster Raptros reported that the database would be transferred to a different server.
Staff then reported "waiting on two new hard drives to come in" to replace failed hardware, which they did on Friday afternoon. Advertisers were given ten extra days in compensation.
Update (15 Dec): An announcement posted on Sunday morning:
The last of the data is importing, and we'll be standing by to finish the upgrades. ETA should be tonight.
Update 2 (16 Dec): Oversized notification tables were cited by FA coder net-cat as both the source of delays and a reason for the slowdowns:
[The current method of notification handling] is not scalable and quickly becomes unsustainable for sites with 10^5 and 10^6 users.
Staff have rewritten queries, tweaked database settings, and intend to prune notifications older than 90 days soon after the site returns.
Update 3 (17 Dec): Fur Affinity came online for a few minutes before stalling and returning to read-only mode, "unable to handle a flood of users while rebuilding the RAID arrays". The site returned eight hours later, with mass notification clearing options disabled.
Users flooded social networks to complain about the disruption and compare alternate sites; primarily Inkbunny, SoFurry and Weasyl. On FA's forums, a 20-page thread was locked after discussion degenerated; it was soon replaced, while fans clamoured for software upgrades.
Weasyl traffic jumped on Wednesday, while e621, SoFurry and Inkbunny, being more active, saw relatively moderate increases.
Conflict of interest notice: This story's author is also Inkbunny's lead administrator.
Other sites appear to have benefited from the interruption, with Weasyl exhibiting a traffic spike, while Inkbunny's upload volume has ranged from 150-200% of normal during FA's read-only period. Meanwhile, guests continue to visit FA, but registered user traffic has fallen in half. [habnab.it]
Site owner Dragoneer fought back against allegations of poor use of donated funds:
I keep seeing that we get tons of donation funding, but that's not true. A few years ago? Yes. But we're not receiving donations. Maybe one or two a month, and it's not much at all. […] I need to update the page again, but most months we don't get anything, yet it doesn't stop our costs from being near $1,800 a month.
He also expressed irritation on finding out about a third-party watch transfer tool, named after Fur Affinity's site mascot, which its creator (and Weasyl) promoted on Thursday:
@weasyl http://projects.weasyl.com/fenderbender/ Really appreciate you trying to kick us while we're down with this insultingly named script. Classy.
The most recent statement, as posted on Fur Affinity early Saturday morning, is more positive:
Data migration is nearly complete. We will be attempting to resolve thins tomorrow (Saturday). Our replacement hard drives arrived as well, and we will get the drives updated and installed before bringing the site online. Updates will be posted to the Forums and via Twitter once we have a more accurate time frame on getting things back up and running. We would like to apologize for how long this has taken. This was an unfortunate issue that we had hoped to avoid.
The drive failures were previously reported to be "not related to the slowdowns or issues"; they were just "incredibly bad timing".
About the author
GreenReaper (Laurence Parry) — read stories — contact (login required)a developer, editor and Kai Norn from London, United Kingdom, interested in wikis and computers
Small fuzzy creature who likes cheese & carrots. Founder of WikiFur, lead admin of Inkbunny, and Editor-in-Chief of Flayrah.
Comments
Children, the lesson of this story is clear:
LOL, InnoDB
Quite honestly, if a furry website made an app or something for the business furry that could be used to set up and track their commissions easier than doing it by journal and then charged a dime per commission to use it, their site would not have to rely on donations, they would rely on those that use it as a way to make money off their art.
The stemming of this conflict with FA is the fact that there are two users. The consumers/social furs and the providers/business furs. The FA staff clearly thinks of their site as more for the former than the latter and because of this conflicts like this will arise.
For instance, the conflict in the thread was a "consumer" fur was assuming that people were upset because of the cost of the site (free), which is the primary force a consumer worries about. The artist responds, by saying the cost is not the problem, the reliability is. So I think that as a furry website is relied on by more and more business furs it is important that they have a methodology in place to keep it in place.
I think creating an app which would make it easier for artists to track their commisions and making some business fur centric programs and then charging a SMALL fee for the service will bring the best of both worlds. Artists can rest assured that the service will always have financing through the work that they do. And the more artists that come to the site (and thus bringing more maintenance/issues) the more of these fees would come in making it work as the site grows.
Unfortunately, I think it's too late for FA to create something like this as I doubt business furs would pay for a site they seem to be turning against when it's free. The newspapers say you can't beat free... actually you can, just don't be greedy and have most of the income go to actual costs of running things instead of going to a board/investors.
I invested 50 bucks in Weasyl after AC 2011, and I'm making bubkis in return. However, it seems to me to have been a good investment.
[comment removed on request]
[comment removed on request]
It's nice that even though the minor drama they caused with the naming of that tool was unintentional, they still had the good graces to change it when called out on it to try to appease everyone.
It wasn't even that insulting even if it were intentional! I think Dragoneer needs to have a thicker skin if he wants to be remain a popular representative in the furry fandom.
No, I didn't think it was insulting either. It was funny! Maybe Dragoneer should have thicker skin (and you'd think he would, by now...) but I think the name change says more about Weasyl than FA. Weasyl is willing to change and work with people in order to keep on good terms with as many parties as possible. It says to me that they want to maintain a positive face and not feed drama. That's not something that can be said of every site out there.
[comment removed on request]
I dont have anything against dragoneer, or pay attention to "OMG dragoneer is literally hitler!!1!" drama in various places for running a fan site at his own expense. I'd treat him as nicely as anyone else, and get him a beer if I see him.
Going to make an assumption about the "got interviewed, didn't even know it" comment: he's referring to the story I put up about Furfling. In it I reposted comments he posted to reddit's r/furry section. Where he was already responding towards another users public post about the topic... and Greenreaper, #1 rep for Flayrah. Yeah, then I came in and asked him to say more. On a public forum he chose to address the subject on first. Dragoneer's comments prompted me to write the story, I didn't bring up the topic to him. (And I said that anything I asked him by personal message was for a news article and he should feel free to say "no comment.")
Check for yourself thru the links here
http://www.flayrah.com/5283/furries-share-worries-about-pay-dating-fraud-spotlig...
if he was referring to another "got interviewed", then I retract this comment, but I'm not aware of any other.
Both parties here have dual roles. Dragoneer runs a site, and also does personal fan activity. I do personal fan activity, and also write stuff to put on Flayrah (which is actually personal, too... hardly trained or paid.) That's the nature of fan communities. OMG "journalistic ethics?" Get off it. Courtesy yes... being the same as NPR or PBS, get real. There is even a recognized school of subjective journalism, and I doubt it's possible to name a better place for it than among fans.
This is, as they say, the Internet. It's best to assume that anything you say will be rooted out and quoted five years down the road - and be willing to stand by it, at least as your honest opinion at the time. People may allow for controversial statements, but they rarely forgive inconsistent ones.
Flayrah offers a range of forms of journalism, as befits its collaborative nature. Of course, it's important to pay attention to certain niceties. For example, whatever the source of a quote, it must be accurate.
By the by, Green...Weasyl needs to be added to Flayrah's profile 'links' section ( http://www.flayrah.com/user/UID/edit/Links ) . It is at least as significant as SF and IB these days, and they are in there.
There we go! (P.S. The link to Weasyl in your FA account description needs 'bunny' at the end.)
The truly interesting thing about that, as you know, is that I did that first set of changes to fix the name issue on that tool. But wait, people say, you're not affiliated with Weasyl.. you're not a Weasyl dev? No, I'm not, but they _DO_ have their external tools in a nice Github Repo, which I forked, and then made the changes and issued Pull Requests. I also found another issue that I took care of as well.
So not only did they get it fixed VERY QUICKLY, but it was fixed BY THE VERY COMMUNITY THAT WANTS TO HELP ANY / MANY FURRY SITES SORT THEIR STUFF OUT.
That's the telling thing. It was an open source fix. When we look at projects regarding FA, we're told by Yak and others that we are "hacking", even for things that aren't touching the server any more than a standard user. When we talk to Weasyl, we're told , "Hey, go for it! Here's our Github, here's a project listing on our website, here's some help, do you want an API? What do you want in an API? We'd love to get help on that, would you like to join our dev channel and hang out and talk to us?"
One of these methodologies will sustain a site a lot better than another.
As Skyler pointed out, FA is free to use, but not free to lose. One could argue that artists are paying their fair share with the art; the rest needs to be paid for by everyone else. Perhaps this was how it worked in the past, but artists have come to bear a greater share due to advertisements, and they expect a return.
You can beat free, but freemium (which is essentially what FA has here) is a powerful nut to crack. If they need funds, it would make sense to expand on it. The problem is they have limited talent to write such features; and programmers may be unwilling to volunteer if their work is restricted, unless they grasp the necessity of an ongoing revenue stream.
If you're willing to forgive the actions of one Allan Greenwald, you sorta forfeit the right to be offended by others' douchbaggery, just sayin'.
Good story but it seems to have jumped past other stories in the queue.
"If all mankind minus one, were of one opinion, and only one person were of the contrary opinion, mankind would be no more justified in silencing that one person, than he, if he had the power, would be justified in silencing mankind."
~John Stuart Mill~
I have stories going back to last May still in the queue, but I don't mind being jumped past because I recognize that this is more time-critical.
Fred Patten
I'm in an awkward situation when stores need to be written on a short deadline; even more so when I've been busy working on other sites and haven't been able to edit. I actually have another story of my own that's just as long which has been waiting over a week, almost completed - but it's not a time-critical subject.
The queue is based on various factors - the priority of the story, its importance, and the mix of content and contributors. If you post two similar stories, the earlier one will probably be put up first, but that's about the most I can say.
But if you extra busy then shouldn't there be another editor. At the moment things are just sitting in the queue and then sometimes we get floods of things posted. Ideally it should be a short delay or one or two things per day. It also looks really bad for me when I interview someone and then have to say, well sorry but it's just sitting on the site for a week waiting to be looked at.
"If all mankind minus one, were of one opinion, and only one person were of the contrary opinion, mankind would be no more justified in silencing that one person, than he, if he had the power, would be justified in silencing mankind."
~John Stuart Mill~
Don't blame him for that; he keeps asking people, and we keep telling him no (and yes he asked me; he already said who he's asked somewhere recently, so that not a secret).
But I will tell you this; I don't have nothing in the queue at the moment. Partially because I've been in a funk since August, and I am three monthly columns down, two movie reviews, two game reviews and approximately a metric shitton of comic reviews behind, plus probably something else, but when I submit stuff, I don't usually have to wait that long. Now, I'm assuming something is in the queue with your name on it, but I don't know how long it has sat there. Maybe we're about the same, actually, and I just am more patient about it; but I know me, so I don't think that's it.
But, are you linking your own stories and putting in pictures and otherwise coding your stuff? You've got enough stories, and they're above average to good usually, so I don't think Green Reaper has to go over it with a fine tooth comb on the writing side. However, if you're putting in your own links, doing that sort of thing, then he doesn't have to do it for you, and that saves time. He may skip over a beautifully written Rakuen Growlithe for a meh crossaffliction because, hey, less work.
If I'm completely off base, I apologize beforehand, because you code your own Newsbytes, so you can make a link. But, you know, you're probably not the only one reading this comment, and this applies to everyone. Also, if your story is time sensitive, you should probably message Green Reaper. If it's not, edit it yourself as much as possible, and then, have patience. It'll get there when it gets there.
Thanks for trying to get me out of it, but the responsibility is mine; either to find and train a backup/sub-editor, or do the editing myself. Worse, Rakuen is good at providing copy and put in some extra effort this time (which I think you will appreciate when you see the story).
My issue is task-switching; it's more efficient for me to do a batch of related work over a few days than to spread it out over the week. Usually I can each thing fast enough that this isn't a huge issue.
To link this back to the topic, this was an exceptional week. Inkbunny saw a 40% increase in traffic, double the normal upload volume, and a flood of new users - and tweets, which it fell to me to handle (Inkbunny's staff contains nobody active there or on Facebook; another issue to address). It was the site's busiest time in the last three years, and I had to be there for it.
[comment removed on request]
It's not really time sensitive but I do get frustrated. It taking a few days is not a train smash but it does make me seem a bit unprofessional because I was the one who did the interview and pressed against huge delays there and the biggest delay is after the interviewee has done all their bits. I also do add all the links and stuff. When I submit to Flayrah it's what I'd accept as a final version. Sometimes some changes get made, usually new links being added. It's also not great motivation to do other stories. I have something in mind as a reply to an article written elsewhere but I know there will be a delay and that really saps motivation.
I do, of course, sympathise with him. I know he's got a lot of work and it's been really busy but it is still his responsibilities to make sure his sites are all working. I suppose it's not his fault if no one will help him but the issues shouldn't have popped out of nowhere. (Except FA going down but that didn't create new problems it just made existing ones worse.)
But, to be clear, I don't have any ill feelings to GreenReaper. I still consider him a friend and my complaints are only intended to try improve things. If I thought he was incompetent or something I wouldn't even submit here.
"If all mankind minus one, were of one opinion, and only one person were of the contrary opinion, mankind would be no more justified in silencing that one person, than he, if he had the power, would be justified in silencing mankind."
~John Stuart Mill~
There's also something that has been a constant with their site's problem: The ad nauseam explanation of that "we are waiting on a new hard drive(s)".
Doing it myself, helping others, or working's with somebody else's off-site provider, getting hold of a new HD was always measured within hours, not days, and even not being an IT professional, the most arduous job I ever took on consisted on procuring 12 drives for a business' commercial website (local Fry's electronics, 30 minutes purchase and return time), which then became three 12TB RAID arrays (2 to 3 hours), which were then filled up at 25% capacity, tested, and hook up to the net within 3.5 hours. With one drive failing to initialize (returned/exchanged), Thai lunch and putzing around, 6 hours,...
So unless these plates are being shipped from downtown Bumf*ck, Egypt, or they already got them, but the data is being physically chiseled back directly onto the platter's surface, their default "hardware not found" error message justification will cause people's eyes one day to get stuck on a 180 degree roll.
Inb4 "freemium!": Yes, quite so, but many a person have willingly provided good fait support in numbers in the past, mostly monetary, so "lack of funds" or "coding/maintenance help not available" is not a excuse to fall back on, plus a lot of artists, fans and professionals, have tied their desire and willingness to make a living through the site exclusively, ergo do quite expect the proverbial unimpeded caca collision with the oscillatory object backlash,... once more.
Tl;dr: "Too many cooks in the kitchen!"/"Everybody called in sick!"=Fine, understandable. The "No Harware and/or Wetware!" proclamation for the nth time, when it has been provided when asked=Eh, not so much to mollify most of the (very patient) proletariat.
The issue that caused the mid-December outage never had anything to do with hardware, even though the FA administrators attempted to move from one machine to another attempting to fix the issue.
This was, start to finish, a software database problem; specifically, the database portion that generates notifications per-user in the upper right corner (E.g. 1020S 105C 41W 2N). Those notifications never expire for any user, and the software backend they are stored in is inefficient, meaning that millions of submission/comment/etc. notifications that will never be read or used have been choking it.
The real solution to that problem is culling notifications, and putting the notifications framework into more efficient software. No amount of hardware will solve that problem. In fact, until yesterday (12/19), the 'Nuke all submissions' button had been disabled because of the high load it was causing the server.
(In fact, based on what I've read, the hard drive failures - particularly the ones on Monday - were probably caused BY the data transfers back and forth torturing the disks. One FAF admin site status post said as much.)
Further evidence that these problems are about software and not hardware: a Friday 12/20 restart. http://forums.furaffinity.net/threads/867077-Server-restart-on-Fri-2013-12-20
I do notice that FA doesn't look like its changed very much since its inception, the layout and style looks very dated. Now I appreciate the mantra "if it's not broke, don't fix it" but the site is reminding me of MySpace circa 2006-2007, making small incremental improvements, but really resting on its massive user base for its future relevance. All it takes is for one of its rivals to break away from the pack (like Facebook did for MySpace), and FA may be on a slow decline to obscurity. Not saying it will happen, but it's a definite possibility.
I see Fur Affinity as a social site from my perspective, not seeing lots of selling, just occasional posts of a piece, and adding to the comments that the artist does commissions.
Myself, I don't like some of the new features on sites, they seem to be there just to look cute or like the latest tech, but they're not as usable. Page content that adds to itself as you scroll down, Firefox can lose the place you were reading, that's annoying. I notice that I'm also having to click a lot more on sites, 'load more', 'view more', and plus signs to expand content.
I think FA is doing things right, with flat quieter pages that encourage reflection with the art or journals posted, not site features like dashboards and such.
The bottom line is that I want to interact with my friends and fellow Furs and their art, not with the site excessively.
Boomer
Post new comment