ZX-VFS Use Case

[Jim]

I am investigating ZX-VFS as the storage mechanism for a document management system. Our strategy is to save documents in a file system directory on the server until a count or size threshold is met. At that point the files would be grouped into a ZX-VFS archive and the individual files deleted. From that point forward when the file was needed it would be read from said VFS archive. The archive and its contents are never written to again. When a new threshold is encountered, a new archive it built. Files are typically a few 100 K but somes can be 3 or 4 mb. This is running as a subsystem in a JBoss 5.1x server.

What I am asking is your opinion as to fitness of ZX-VFS for this task.  I realize that you stated it is not thread-safe, but is that still true when the archive is only opened once (VirtualFileSystem object shared amongst threads with the potential for concurrent access) and is only ever read. It is never written after its initial creation. Also what are the practical limits both in file count and total file size of an individual archive. Is there any limit to the number of simultaneous archives opened?

Aside from any issues with the threading you may tell me about, it works great. It took me only about 30 minutes to create a plugin for our document management system and it ran great the first time. Great job!!

Thanks
Jim Moring

[ZipXap]

Hi Jim,

ZX-VFS is the back-end for ZxApp IT.  I have used it to create archives that were several gigabytes in size, with hundreds of thousands of files.  I do have a minor update to it that I have not put online yet.  That said, the threading issue will still affect you.  It uses a single instance of a Java random access file, so all threads would be fighting over the file location indexer.  I don't have much time these days, so I can't make any commitments to it, but I will try to revisit it this weekend to assess how easy it would be to make it thread safe... That was always part of the plan, but since it wasn't immediately necessary I didn't worry about it.

Kurt

[Jim]

Hi Kurt

Thanks for the reply. If you could scope out the changes that would need to be made inorder to make ZX-VFS thread-safe from a read-only prespective, I would be glad to work on implementing the changes. As you do not need it to be thread-safe for your use case, I do not need it to be thread-safe except when being read in my case. Therefore I suggest limiting the change to read-only thread safety. Let me know what you think. I will start to familiarize myself with the source.

Thanks
Jim

[ZipXap]

Hi Jim,

Sorry for the delayed response.

After some investigation it looked fairly easy to do, so I went ahead and made the changes necessary for ZxVFS to be thread-safe for reading.

I had to do the following:

1. I found all usage of the RandomAccessFile.seek() and RandomAccessFile.getFilePointer(), and made sure the usage was synchronized so that threads don't alter each other's the file pointer location.  (There were many places)

2. I created a VFSSectorPointer class to contain the current read/write indexes for the VFSStreamReader and VFSStreamWriter.  I had to do this because VFSSector is supposed to exist for the life of the VirtualFileSystem, so obviously it couldn't contain the current read/write indexes.

3. Also, unrelated to making the virtual file system thread-safe, I found and fixed a bug that was preventing users from re-encrypting the VFS.

I will be uploading the fixes and enhancements later today.

Thanks,

Ken