http://www.zenphoto.org/support/topic.php?id=10466

Wow, so finally I concluded that regardless of when it's done ImageMagick in zenphoto crashes my server seemingly due to abundant memory usage by parallel processes where image magick by hand (convert) works fine and also in G3. and I get... "go to G3... thread closed" I was at least expecting "so resize by hand"

I wasn't rude or demanding and even said it's one of those things where ok, maybe for now that's how it is. But I would think the dev would say, "gee if our multi-threading crashes someone, that's a bummer. It's corner case for userbase(maybe maybe not) though and probably not in our time and scope to sort it out now though." It seems the devs here don't want any critical feedback and take it as insulting. Actually, I spent far more time(not that it was much) trying to udnerstand the problem-- downloaded the beta from svn, investigated code differences with how G3 does conversion, and would have investigated more-- than the time I was trying to save by using what seemed like a more user friendly program. Why? Because I'm trying to be insulting? I read a review about ZenPhoto where someone ditched it for this very reason. I guess now I get it. Feedback isn't insults. If you don't want feedback don't open source your code with a public forum.

Maybe this will get me banned. If not, the funny thing, I may keep looking at it as I get a chance and if I find a solution I'll post it back. I respect the work done here in spite of the odd responses.

Well, perhaps if you could be more suscint in your postings. We really do not appreciate waiding through volumous llocorhea which is mostly pointless. BTW, you were both rude AND demanding IMHO. If you are trying to make me feel guilty for not being nice to you, you are beating a dead horse.

Zenphoto is a volunteer project, so the customer is not always right.

We really do not care how G3 (or any other gallery "does it".) If those are better for your needs, be gone. If they are not then be straight to the point.

As to Imagick crashing your server. Neither (yes, there are only two of us!) of the developers has access to a server that supports Imagick so cannot even test. If your endevors find somehting that can be corrected please submit the change.

It doesn't appear that imagemagick does crash the server per se. Convert works. G3 works. Both are imagemagick. MAYBE there's a point that both use convert and not the php interface (maybe even without php experience I can modify zp to test that) but it appears far more likely that ZP is creating multiple processes and that's the difference. Since you don't have access to a server that uses ImageMagick and I do (bare metal access), I'm even MORE surprised at the impatience.

"We really do not care how G3 (or any other gallery "does it".)"

Ok, that says a lot when theirs works and yours breaks. It could turn out it's something about my server; that's always a possibility though.

You're volunteer I get that. If you aren't interested in developing ZP, don't. If you just can't address this issue, even possibly with input, say so. No need to get snippy.

sbillard:"Maybe you should actually look at the product before you make judgements."

Case in point about snippiness, maybe you should actually look at my judgement before you tell me what maybe I should do. I did go ahead and use the product and the judgement that it doesn't automatically detect what theme is installed and thus what images it needs (at pre-cache time) held up just fine (maybe I overgeneralized guessing that there was NO interface to the theme ). Sorry for trusting the description (which was correct).

[i] It seems the devs here don't want any critical feedback and take it as insulting.[/i]
Before this ends in a unnecessary flame war. We appreciate any feedback and of course critism and we don't just ban someone... If you look around you will see that we help a lot if someone asks and often more than on other forums...

But please understand that your quite long postings sound a little too demanding, sarcastic and insulting a la "there it works and with our 'crap' software it doesn't". Note, that we are a much smaller team than G3 (which you used as a reference). Try to be a little more objective. Common issuse that written text is easily missunderstood. Maybe you just wanted to provide feedback but it reached us on the wrong foot.

Of course we cannot really care about any other tools and how they may work. IMHO Zenphoto is focussed more on personal portfolio websites than huge pure image galleries as G3 might be.

Once again: Zenphoto does not know about images sizes needed on a theme (unless ituses the default option values) until the page is requested. You have the flexibility to create a layout with any size and changing on every image if needed. The only way to make those precachable is to register the sizes as mentioned on the other topic.

Also, the imagick support was done by a co-developer that currently lacks the time to work further on that. So there are some thing that might not be right or optimal. That is the nature of software. There are a few tickets open and postponed for that reason). I do have a server (well, shared hosting) with Imagick but I don't use that much and big images on there.

If you want to help improving the code, you are invited to work on that and/or open a ticket to make suggestions. Open also a ticket about the crashing with all technical details. Maybye Our co-developer can look into those later. As said we are smaller so these things might take a while. That is the way to do it.

I apologize that I'm only now chiming in, I haven't seen these posts until now. If you have a crash log/stack trace for the crash, please do post it and I will take a look.

If it's crashing for memory issues, though, then that's not something that Zenphoto or Imagick actually handles beyond the very basics. Regardless, there is a memory limit option for Imagick (which is exposed as a Zenphoto option) that could potentially alleviate your problems. It will cap the memory usage of any given Imagick process, the default being unlimited, meaning that it will leave it up to the server to manage the memory. (see http://us3.php.net/manual/en/imagick.setresourcelimit.php)

Please be aware as well that G3's implementation of ImageMagick differs from ours. G3 directly invokes convert via PHP's exec, while Zenphoto uses the Imagick PHP extension which exposes ImageMagick's functionality directly to PHP. (I saw that you already know this from your other post, but it's good to put here in case someone else stumbles upon this in the future.)

The reason for the difference is that ImageMagick itself is fairly ubiquitous whereas Imagick, while being available on many hosts, is not as readily available to everyone. It's a trade-off between code "correctness" and a more hackish, yet absolute way of handling the images.

In regards to httpd processes using both memory and disk I/O, processing images will necessitate loading the entire image into memory, which can be very resource intensive. During processing, Imagick will use the temp directory set by ImageMagick's MAGICK_TEMPORARY_PATH, memory mapping images to the filesystem, thus the I/O.

One more thing: the way that the Zenphoto image processor works is that each image will generate a separate HTTP request. So, based on server setup, each image will spawn a separate process.

Thanks for chiming in!

@kagutsuchi:

"One more thing: the way that the Zenphoto image processor works is that each image will generate a separate HTTP request. So, based on server setup, each image will spawn a separate process."

Thanks for the reply. You hit/reviewed many relevant points, but I think this one is probably the key. I know about the imagemagick memory limit, but it doesn't help (in the end converting large images takes large memory). I doubt the convert vs php interface is the problem either, but I think we both recognize it's a difference so it is something to be ruled out possibly. As far as I can tell both interfaces behave pretty much the same though.

For whatever reason G3 seems to serialize more and that's the difference.

For me I think what I would need for ZP to work is a precache mechanism that serializes, limiting processing to one file at a time without needing to globally limit the number of httpd or process on my server at all other times.

In the other thread I made some other comments regarding ideas on how themes could register all their caching needs (even as the theme is used and configured in different ways, for anything but truly on-demand image sizes) so the cacher can be aware of what images to produce. This is a more general idea about making a precacher better integrated and work more easily (probably for developers and users in the long run), an idea which evolves naturally once a specific use case like mine really requires a precacher.

However the core point is that a use case like mine does actually require a precacher, and specifically requires a strictly serialized precacher. As it turns out, the parallelism problem exists in the precacher too.

here's the ticket as requested:
http://www.zenphoto.org/trac/ticket/2260

It is impossible to turn off on-the-fly caching without either major architecture change to Zenphoto or implementing some manner of IPC. The on-the-fly caching is considered a feature of Zenphoto and it is unlikely that this will change in the near future.

Performance is always a trade-off between memory, CPU, and disk. In an environment as you have described, memory and CPU are the first resources to be exhausted and the parallelism exacerbates the issue. It's typically not a bad thing to max out the RAM and CPU as long as it isn't causing problems.

http://www.imagemagick.org/script/resources.php#environment

In a threaded httpd environment, it might be possible to use MAGICK_THREAD_LIMIT to enforce the max number of concurrent threads. In a standard httpd environment (that spawns processes, not threads), limiting threads alone will not work. A better solution might be to use MAGICK_THROTTLE. Doing this, you can be sure that any given caching process will, at convenient points in the processing, yield to other processes that are vying for CPU time.

As for the precaching mechanism, the best solution would be to have the user select how many images are precached at a time (with a sane default). In your case, you could choose to have one image cached at a time. So, at any given time, you would have one image being cached plus however many are currently being cached on-the-fly. Once all images are cached, it sounds as though the server can handle concurrent requests.

To me, this is really the issue here: the precaching mechanism can be too resource intensive for some servers. As last I recall, precaching is considered unsupported, so it's really up to the devs what to do from here.

Given that fact, and the fact from your ticket that you are using uncompressed bitmaps, I think the most prudent advice at the moment would be to use processed images on your site instead. That will help alleviate resource issues and you can provide the uncompressed version of the images outside of Zenphoto.

@kagutsuchi:
[i]As last I recall, precaching is considered unsupported, so it's really up to the devs what to do from here.[/i]
Since Zenphoto is normally used for "normal sized sites/galleries" so it is considered normally not necessary. Bitmaps file types are also no "standard web format".

Caching is actually supported because we got a lot of requests to it because similar reports. We have a cache_manager plugin now.

It is actually quite intregrated but it requires the sizes set manually if you set custom sizes on the theme directly which always overriding backend settings and are only "known" to the theme otherwise (a great flexibilty feature IMHO if you need fixed sizes or maxspace images for whatever layout reasons), You can even let your theme (or plugin) register sizes to the cache manager. Examples are found on the themeoptions.php of standard themes like Zenpage.

Sbillard is the better person to respond on the technical site but unavailable until around the weekend.

Ahh, learned something new today. Thanks for the heads up

"Given that fact, and the fact from your ticket that you are using uncompressed bitmaps"

I'm not using uncompressed bitmaps. I'm just using full size jpegs. However when imagemagick resizes, it must uncompress teh image to a bitmap (at least one copy) in memory and it seems that's exactly what happens because a 5MB jpeg using soemthing like 100MB of memory as I recall during resizing. That's fine for one or two at a time. For me, 10 or even 5 at a time is not fine.

That's the crux of the problem. I can serve out 5MB images on my bandwidth very well and that doesn't use that much memory. I cannot have imagemagick uncompress a bunch of them though.

Yes figuring out how to make the precacher do one image at a time would solve the problem

...other than the "and whatever images are also being cached on the fly" issue. Once things are precached, and lets say we add one more image that's not, I'm sure I can disable on the fly caching by commenting out the call to imagemagick in one line. I don't know if some page will cause errors, but that's even ok for my case.

However when kagutsuchi you say on the fly is so integral to things, I think for this purpose what this really means is that even the precacher is really using the on-the-fly mechanism and is just asking for a url for each image or page and letting the on-the-fly thing do it's work. So what it really means to say it's very integral and cannot be disabled is that someone would have to actually make the precacher build the resize command for each image on its own, without quite as much code reuse, but still some. Once all the iamges are there, commenting out the on-the-fly call to the resize will certainly not break anything since that code branch won't even be reached, so it's just a matter of getting them there in the first place.

oh and I'll look at some of the thread limiting options mentioned. I haven't been able to find one that works well previously.

You cannot disable on the fly caching. That is the only way Zenphoto creates the cached images. There are several reasons for this, but the bottom line is that it is the only way things will work.

You should be able to configure your server to limit the threads that a web application can use. That is the simplest and most effective solution.

Absolutely you can. If the cached images are already there, then it doesn't do anything. I'm quite certain I can go comment out the line right now with a test case with no problem. I get that "it's the only way they are cached" in other words right now the precacher is using that instead of making it's own call to scale.. and that's my point, it would require a slightly more independent precacher. I think it can still reuse some of the code though from what I've seen.

Of course you don't actually comment it out to do it right, you wrap it in a conditional and return one of those broken image icons.

I've actually done quite a lot of programming at the under the hood level, not user interface level, so I'm not just talking from the perspective of how I think the world should be. I just don't know php or uncommented code. I have looked through the code enough to have a little clue though even regarding this specific code.

There is already as I recall a conditional that says if the image is not there, then "precache" it. You add one more conditional inside that says and if disable_precaching is true then return a default blank image or broken icon or page error or whatever level of elegant failure you want (even do nothing and rely on sanity checks/errors/failures elsewhere).

The only issue then is satisfying the first conditional.. ie making sure the all the cached images are already there. And so again it comes down to making a precacher that doesn't just leach off the on-the-fly mechanism. It can still call the same code snippet to do the resizing though once it gets the information it needs about the image name and size requirements. It just can't do it by simply requesting to GET that image.

Be my guest. Obviously you understand the code better than I.

Haha, but you said you cannot disable on-the-fly caching and it's just not true.

I get that on the fly caching is beloved by the developers but here are 4 facts I hope we can all agree on:

a) if an image is already in the cache directory (and we're just talking about a file in a directory) it won't be cached on the fly when it's next requested and the Imagick scale call will not be made.
b) It is possible to put an image in the cache without using your on the fly code. In fact it can even be done by hand or by bash script so long as the person or script has access to information about filenames and sizes needed as the precacher does.
c) If the on the fly code (or me personally) can get that information at precache time then another code also can.
d) Even if an image is not there when requested, failure is an option. And well that's obvious: if you use precaching only and you don't precache something, then a request for the image fails. That's true in any pre-cache only system but that's like a "der" kind of thing that is obvious to the user when they select the option to disable it.

Please tell me which of these 4 statements is not true.

by the way I can see that maybe in a slide show or soemthing failure is less pretty, or harder to make pretty, than in a static html file with an href link to a missing jpeg. But failure is failure and is expected behavior when a file isn't there and there is no mechanism enabled to produce it. The only thing to worry about then is maybe how gracefully to fail. I certainly don't know that part of the code, but I don't even care how it fails. Certainly it will manage to fail somehow without any effort by the developer, and that's perfectly fine with me graceful or not. Gracefulness is a bonus.

Here's a mixed/dumb syntax psuedo code:

global bool on_the_fly_enabled

Show_image(image scaled_image, bool precache){
If scaled_image exists then
display scaled_image
elif on_the_fly_enabled==true or precache==true then
resize (image_source, scaled_image)
display scaled_image
else
burn in magnificant flames of destruction! (or fail gracefully)
fi
}

Precahcer(){
For i in all_scaled_images_list {
show_image(i,true) > /dev/null &
c note the metaphorical ampersand above should be removed
c to serialize this
}
}

Notice that I even make a precacher that DOES use the on the existing on-the-fly code(I'm sure it's not necessary but it sounds like that's the case presently), but with minor modifications to add the on-the-fly and precache booleans in that on-the-fly code.

Maybe the precacher works on a page by page basis or whatever, fine.. it still just needs to set a flag with some scope limited to it's own process to enable the "on-the-fly" code. The philosophy doesn't change. I would think it could just call the bit inside the conditional directly but I'm trying to be a team player and work within the philosophy of always using the on-the-fly code.