I thought some of you might be interested in what I've been working on the
last week or so. Basically, I've changed the way squid stores files. I
think it'll be a big win, but I haven't run any real tests yet. As far as
I can figure, worst case is that it won't be any faster than "regular"
squid, but will greatly reduce the number of file descriptors used.
I just got it running smoothly this morning, and I'm letting it run
here on my server for a few days. After that I'll probably be testing it
on a production system to give it a real stress test. If everything goes
well then I'll release the code.
I'll just attach my little README file here.
----
Notes for squid-1.2-SQUIDFS
Instead of storing each object in a seperate file on the filesystem,
all objects are packed together into one file. There are several reasons
for doing this.
1) Greatly reduced usage of file descriptors. Using the old method, squid
used 1 FD for every object that was in transit. This got to be a
problem on busy caches, since FD's are a limited system resource.
SQUIDFS uses only 2 FD's per cache_dir.
2) Improved speed. Old squid was opening and closing files frequently.
File operations like that are very expensive system calls. SQUIDFS
opens its files when it starts up, and closes them when it shuts down.
It merely needs to do a seek to the appropriate place for each read or
write operation.
Old squid used to store the filenumber that its data was stored in.
SQUIDFS records the file offset within the swap file. When it needs to
read or write data, it just seeks to the appropriate offset to perform the
operation.
Some notes:
- in squid.conf you specify the cache_dir(s) with this syntax:
cache_dir <swap file name> <log file name> <size>
ie
cache_dir /squid/cache/swap1.data /squid/cache/swap1.log 100
would configure a 100 megabyte swap file called /squid/cache/swap1.data
Note that you can put the swapfile and logfile in different locations!
Also note that you could use raw partitions to hold the data!
ie
cache_dir /dev/sdb1 /squid/cache/swap1.log 100
would create a 100 meg swapfile on the first partition of the 2nd scsi
disk (this is a Linux example...YMMV)
NOTE NOTE: If you do this, any data that already exists on that
partition will be destroyed! Only use partitions that you have
specifically created for the purpose of storing swap data.
- there is no way to rebuild the cache if squid is not shut down properly!
If there is not a clean swap log present when squid starts up, it will
start fresh with an empty swap. I'm pondering ways for rebuilding from
a dirty swap file...it may show up in the future.
If you just send squid the kill signal (ie 'killall squid') and wait
for it to exit properly you will have no problems.
- space in the swap file is allocated in 1000 byte increments. Thus, if
the object is 1453 bytes, squid will allocate 2000 bytes from the
swap file. Yes, its slightly wasteful, but reduces fragmentation of
the swap file a great deal.
- more anti-fragmentation code will be added later.
- this code has only been tested on Linux so far. It SHOULD work fine
with anything, but hasn't been tested yet.
Feel free to contact me with questions or problems.
/-----------------------------------------------------------------------/
/ Eric Stern - PacketStorm Technologies - (519) 837-0824 /
/ /
/ WebSpeed - a transparent web caching server - available now! /
/-----------------------------------------------------------------------/
