Symphony.

Hi Michael,

From a high level perspective it's a site that has many image galleries, curated in the backend from a central pool of imagery.

When creating a gallery entry the backend user needs to be able to upload, sort, order and select anywhere from 5 to 30 images from this pool - and the pool will be ever growing. The original method to achieve this functionality was to have an images section, and a gallery section; the gallery section contained a subsection manager field which allowed the user to interact with the 'images' section.

Over time this crawled to a halt and functionality started to falter, uploading didn't work any more and attempting to search caused the site to crash.

We've also tried using the SLF+ extension with similar results to the above - many of these issues were looked into by Allen and the Soario chaps (who have been helping out with this project) - and although there is room for performance improvements on that extension we won't be seeing them for this project unfortunately.

So the Filemanager is a last ditch effort really; naturally changing the structure a little, instead of relying on a linked section, instead relying on managing images on the server directly (I actually prefer this as it allows for more efficient management of the images via FTP).

In short if I have the field set to interact with a folder with 100 images in it, it works fine. 3500 images though involve 2 minutes of hang before it manages to display the files - and the same wait again after finishing an upload while I assume it refreshes the list. Basically very similar behaviour to the other two extensions, except that it gets there in the end rather than giving up.

--

All that said, if you can think of a better, or even comparable approach to the same problem, I'm all ears - at this point we just want to deliver something that works, but we are already looking at ways to re-build with scalability in mind.

Also worth noting is that this is all in a pretty standard server environment - basic dedicated box - nothing fancy.

I don't quite understand where the bottleneck is. (As I said, I don't know the Filemanager.) Maybe the problem is building a very big file listing in the backend? (Does the Filemanager use pagination at all? Maybe it can not, due to its concept?)

When I have to handle large datasets and relations, I always rely on the SBL field (simply because I consider this a stable and future-proof solution). If data must be manageable in the backend, I often modify the workflow (on edit pages) with HTML Panel fields and a basic "search" (Publish Filtering extension). (Well, that's not very elegant, so I prefer building things like this in the frontend — which requires Members, of course.)

I can't give you a working solution, cause I still have no idea how images in this pool are searched for and connected to entries. (Are they tagged? Do you need a preview? What is the amount of meta data? How many image "entries" must be displayed on one page?) And are you sure that all these authors must work in the backend?

I hope that you can find the conceptual problem and work around it. It's definitely hard to believe that crawling the filesystem takes 2 minutes for some thousand files. You should test that in order to find the real problem.

Thanks for the response Michael.

"I hope that you can find the conceptual problem and work around it. It's definitely hard to believe that crawling the filesystem takes 2 minutes for some thousand files. You should test that in order to find the real problem."

I won't pretend that it makes the most sense to me either tbh, but it kind of is what it is - unless someone with more experience with this extension also dealing with thousands of files can confirm otherwise, I have to either assume that it's a limitation in the extension, or that it's using an exorbitant amount of resources for what it's doing. Incidentally there are no pagination options that I'm aware of, from what I can tell it finds every file and folder from the designated root, all on load. I was hoping it'd stagger it by folder (like FTP) to ease things, but unfortunately that doesn't seem to be the case.

The standard select box link field certainly works, but doesn't provide a particularly usable interface in this case - selecting multiple images from a pool of thousands is never going to be easy in that format IMO - especially as they're photographs and often have rather arbitrary files names - unless there's some magickry that can be had with the standard extension. SBL+ was promising, but refused to handle re-ordering and a couple of other minor issues with this data set - but we had a few of the Soario chaps looking at that, as well as the extension author, so unfortunately that one's a dead-end for now (shame because it performed well).

Are they tagged? They were categorised when they were handled with the 'Images' section, but naturally with this extension they're just images in a file.

Do you need a preview? Some kind of preview, yes, important for reference.

What is the amount of meta data? As with the tags, this is moot when using this extension - but previously it was pretty minimal (author, date, category)

How many image "entries" must be displayed on one page? All of them need to be available for selection, if you mean in the context of pagination then not too many, maybe 30 or so at a time would suffice.

Managing this from the Front-end is certainly an option - and this may be something worth investigating moving forward - unfortunately outside of the scope for now though.

Hi Nathan.

First of all, thanks for trying Filemanager.

To figure out potential bottlenecks you may monitor the response time of the directory listing request (the request url should look like http://domain.com/symphony/extension/filemanager/listing/?field_id={$field_id}).

My guess is that the performance problem in general is frontend related. There're basically two things that happen behind the scenes:

1): parse data from server and put them in a collection. A top level collection represents all directory objects. Each Directory is a model that has its own collection of file objects (also data models) and a pointer to its parent object (parent directory). Each View object, which is basically the DOM representation of a file or a directory object that responds to changes made on a file or directory object (delete, move, select, etc.)
2): when all data is parsed and the top level collection populated, the view objects will render to the DOM. Given a huge amount of file this can take quite a lot of time. This would also explain the UI lags you've described.

Hi iwyg,

Thanks for the info!

To give you some more insight into what I'm seeing I'll describe the loading/response situation as accurately as possible, now that I've monitored it specifically it's quite curious.

• When the page first loads, the field's 'spinner' loading animation begins, and the request itself is 'pending', so far so good.
• This continues for 1.5 mins.
• The request completes after 1.5 minutes with a size of 1.14mb, once complete the spinner loading animation freezes and the site hangs, for around another 1.5 minutes.
• After this tense wait, pop, everything in place where it should be, and perhaps most importantly, everything works as it should.

So the troublesome bit seems to be what happens AFTER the request is finished, I assume this is the rendering and what you mean by the problem occurring on the front-end.

Is this just a limitation or something that can, potentially, be improved?

Thanks

Filemanager updated to version 1.0.8 on 21st of February 2013

hi,

first up: great extension, it helps quite a lot when one needs to manage large amounts of images.

that said, i encountered a problem where i lose all references of images in the fields and xml. i have my install (including latest db dump) in a git repo. after cloning the repo and importing the db dump on another machine the loss occures. i have tried this several times now, always the same outcome. the entries in the the database seem to still be there but as mentioned the fields are empty.

any idea as to where the problem may be?

thanks a bunch :)

Just an assumption... if you are doing the import through git & taking a DB dump, are you making sure that your upload folder with all the images is being cloned? As most of the time this would not be tracked since they would be uploads. This might lead to problems with the files not being found.

no, the upload folder including the image files is included :(

To be honest, I have absolutely no idea

can anyone else confirm this? or is this just my setup?

Just to be clear. By saying you loose all references you mean selected files?

yes. i have a filemanager field images in section product. i have several entries in that section with a bunch of images selected from the filemanager field. all is well, the xml is fine.

i then add all files (including latest db dump) to git, commit and push to my server repo. from my other machine i pull the latest changes and everything goes as expected only that all filemanager fields are now empty. database tables seem to be fine. weird...

i think i just found the problem. while looking at the db tables i noticed the file paths having forward slashes as well as backslashes in them. this seems to be fine for my windows machine at work but my mac at home will not be able to access a file with a path of /uploads/images\imagefile.png right?

can we not normalize the path on save?

let me get this straight: your dev environment is windows and you deploy to a unix system?

Filemanager is using the DIRECTORY_SEPARATOR constant for path creation. Having mixed separators is weather a bug in Filemanager or parts of the path are derived from a Symphony Constant which is not just normalized all separators to / (or just ignores it, I don't know). Maybe normalisation could be the way to go. Since I don't develop for windows systems in the first place (at least not for OSS projects) it's kinda not on my agenda. I hope this doesn't sound arrogant. But thanks for the hint.

yeah, some people just dont have the luxury to choose their working environments freely :) i am forced to work on windows locally at work. i choose to work on mac os at home and my live server is running ubuntu. sure, i could do all my work directly on the server but i enjoy working from my notebook and the point is really, in a decentralized multi-user environment someone will always be using a freaking windows machine :D

that said i can perfectly understand that you dont plan to give up your spare time working on windows opts for a open source project ;) i still think this is a really great extension and just wanted to say thanks again for putting all the work into it.

i'll see if i can fix this or mod my setup somehow.

vagrant and puppet to the rescue :).

Well first of all, you could write a small script that would normalize all paths within the db. I'll get my head around this later. It shouldn't take too long to fix I hope. Heads up.

vagrant and puppet to the rescue

shame on me, never heard of it. reading up right now :)

you could write a small script that would normalize all paths within the db

the thought crossed my mind but then i thought it would be cleaner to solve this where it occurs. but i am still a noob very much when it comes to complex extensions, so every help is appreciated :)

the thought crossed my mind but then i thought it would be cleaner to solve this where it occurs.

Sure, but that wouldn't solve your current problem.

Check out the latest commit on the master branch and please report back if you experience any further issues with mixed path separators. Thanks

Sure, but that wouldn't solve your current problem.

right, i will have to modify existing db entries ;)

thanks for the commit. i will try first thing tomorrow morning! i'll keep you posted.