Re: [tlug] distributed file systems

[Sach Jobb <sach@example.com>]

Hello and thank you for your thoughtful reply.

> So, if I understand you correctly, you have a central server with a
> database and a growing set of files, some of which change from time to
> time. The database is always updated through a web interface running
> on that server.

That's basically correct only the db is on a separate server from the
web server.

> Is it the same case for the files? That is, having
> read-only replicas of the files will work for you? Or do people need to
> be able to edit the files elsewhere and send them to you somehow? How do
> people add and edit files?

We have considered a solution of splitting the input/update/delete
processes from the read process on the software level. So that the
local client would read everything from a local "read only" server,
but anything that required an upload or delete would directly access
our data center in Tokyo. In fact, that's sort of the fall back plan.

The real issue here is performance. It's just too slow from the UK
(NYC seems to be a bit better, but still very slow). If it's too slow
we are justifiably afraid that people will grow frustrated and won't
use it. Before anyone asks: yes the site is heavy and that's because
it's packed with media and there is a little bit we can do to improve
it but fundamentally the business requirements demand that the page
remain fairly large.


> So delays of, say, five minutes are too long. How many seconds delay can
> you handle? And is it practical to ask users to run an update command to
> get the latest changes before accessing the replica of the files?

Good question. Shouldn't be more than 10 seconds or so. This is just
to prevent people from noticing a lag in updates in certain content.
That's why I really want it on a per file basis. Varies a bit between
the systems, but the main system basically accepts all of the data in
sort of a wizard like format, and then pushes everything at the end.

> What is the latency and bandwidth between the replicas and the master
> server? What is the typical size of an update (in bytes), and how
> frequently do they happen?

I haven't tested the latency between the servers yet but this is not
an isolated issue. When we get one working there will be at least one
other local deploy this year.  The fast majority of the files are
images which very rarely get above a couple of MB. The text files are
insignificant and I believe we capped the music files to something
like 100MB. We are looking at moving the MP3's to an external service
as well so that could be a non-issue in the future.

> Sorry, I thought you had only one server, but this implies you'll have
> multiple servers running your application. Can you clarify this?

Yes, there are three right now but at the moment the web app only runs
on one. We will also be adding another tokyo local app server in the
near future.

> You yourself said that you change text files. So you've got versioning
> right there.

This is literally less than 50 files. We needed some way for some of
the content to be changed that did not involve a database for a couple
of customers in some rare circumstances and this horrible hack is the
result of that.

>> ...but wouldn't it be sexier if I just new that every write was being
>> written to the other servers, and the was something in place tracking
>> this against the nodes?
>
> Again, I'm confused about this multiple servers thing. Can you describe
> again where data enters the system and how?

The local servers are really for just for regional deploys
(specifically UK). Most of the data is entered by a few different
groups through three different administrative systems. It's updated
realistically probably by a few thousand people basically around the
clock, although it's heavier from around 14:00 to 20:00.

> At any rate, there are lots of sexy things you can do, but if you're
> just trying to solve a business problem, rather than be cool, you
> probably don't want to do them.

Yes! The business requirements outweigh the sexiness.

cheers,
sach