First, ImageMagick will not decompress a compressed image. This is actually quite impossible without some highly advanced and slow algorithm. Compression is a function executed at exportation and is irreversible without quality loss.

Second, an image's size in memory is dependent on the dimensions of the image, not the filesize. These links give an overview of what goes into loading an image into memory:

http://imagemagick.sourceforge.net/http/www/FAQ.html#C9
http://www.imagemagick.org/script/architecture.php#overview

Third, here's a quick synopsis of how Zenphoto's image processor works. When a Zenphoto page attempts to display an image, it will use something like:

``

For each `tag, the browser will make a separate HTTP request.i.php`, the image processor, will return the appropriately sized image either from the cache or as soon as it has been processed.

This is not a process that can be switched on/off with a boolean as the entire architecture of Zenphoto has been built around this philosophy. It takes advantage of how browsers and web pages work by offloading the crux of processing to a separate process to be displayed once it is ready (also known as lazy loading).

Gracefulness may not be an issue to you, but it certainly is to the average user. There should never be a use case in which an image which could be displayed is simply not. It's disingenuous to call it an error and it's negligent to say "too bad".

The real issue here, in my opinion, is that the full-sized images are simply too large for your server to process. It's like trying to run the latest and greatest game on a 90s Compaq - it might run, but it will never be fully playable. You can massage as much performance out of it as you want, but it will never be optimal.

If you can find a way to make it work for you, then that's wonderful and we would love to hear the solution so that other users might benefit from it! You could even write your own precacher plugin from existing code. It's entirely possible, but it does not sound as though this is a route that the devs will put into development.

Ok.. I am working on a solution (keep reading) but first:

These responses are getting a little strange.
Every image that is displayed is decompressed. Monitors do not know about discrete fourier transforms. They know about RGB pixels and that's bitmaps. All image maipulation tools short of a few things like jpegtran that can work directly with DCT coeeficients.. decompress images. Most web servers don't manipulate images though, so most don't decompress them.

But never mind that, regardless, the fact is imagick uses over 100MB of memory to manage a 5MB jpeg and that's an observable fact. So if we disagree about decompression lets just leave it that we do and I'll get back to the important point and my results

Whooh.. on to the important stuff, ok I made a partial solution so far as you suggest and am sharing as you suggest. I just did this to your code:

function cacheImage($newfilename, $imgfile, $args, $allow_watermark=false, $theme, $album) {
}

Guess what? It works fine. Nothing dies catastrophically. All pages work just fine but show the usual broken image icon when images are not available, exactly as expected. Even the slide show page operates but with broken image icons. The web page loads, the layout is fine, nothing burns up in flames at all. Wow.

So... yes you CAN disable on-the-fly caching.

Three things:

1. Defaults
2. Errors when errors are
3. Admin choice

The devs like a content providing system that also has ABILITY for automatic content creation and that's probably great for some (obviously not for many), but one VERY simple change (that I just proved) can allow other users to enjoy your wonderful content providing system without using your content creation system. It doesn't have to be the default, but it can be an option (admin choice). But content providing takes FAR less resources than content creation and not everyone needs both or at least not on-the-fly.

Almost every content providing server on the planet gives an error in some form when linked content has not been made available. It's weird to act like it's the end of the world to your high expectation user base that if they CHOOSE to turn off automatic content creation and do not pre-create the content, that they would then somehow be massively turned off by a very expected and appropriate error, the same error response that most every web server in the world would give when content does not exist.

So long as all images exist your content providing work beautifily with on-the-fly creation disabled with a one line code change. With on the fly creation disabled, it behaves exactly as one would expect when content is not available too. I can't see why you keep acting like this one very well modularized routine is critical to the working of every line of code in your project. IF that was really true you would be the worst programmers in the world. But it is not true, as I knew, and as I have now proven.

I have not tested every theme and every plugin. Maybe one will give an error in a slightly less graceful way, but an error is still expected and an error is what it will return. Most will probably just produce broken image icons though as such sane behavior is I think built deeply into the html/php tools used. Other functions, IF ANY, that produce slightly less desireable error forms can be worked around probably with a three line sanity check somewhere.

The full size images are not too large for my server to "process". I can serve 40 at time. I cannot resize 3 at time.

As said in the ticket response.. disabling multi-threading on my server is dumb becasue multithreading is used in many ways. the argument was used as a reason NOT to serialize content creation, but it's exactly the reason why serializing instead of reducing server threads IS the right solution.

There is no reason to require parallel content creationg level of resources to allow people to use your great content serving software. They are two separate needs with different resource needs and it's trivial to actually separate them.

"that's probably great for some (obviously not for many)"

I meant that there are also many for whom on the fly creation is not great. It came out wrong.

(I apologize for the length of this post in advance.)

Okay, some further reading reveals that, yes, ImageMagick does decompress JPEGs in memory. This is necessary as image manipulation requires the raw image data. Quality is what is irreversible.

You did not read my links on images in memory. Let me pull the most relevant part here:

Quote:The amount of memory (in bytes) required to store a single image in memory may be calculated via the equation (QuantumDepthRowsColumns*5)/8. Performing an image processing operation may require that several images be in memory at one time. In the case of animations, hundreds of images may be in memory at one time.
Image processing necessarily uses a great deal of memory no matter how you go about it. This is why machines dedicated to high resolution image generation/manipulation are typically loaded with RAM. You simply can't expect the same performance from an outdated machine; it is ill-equipped for image processing. I'm not sure what the complaint is here.

The idea behind on-the-fly caching is that it is used only when needed. If the cached image exists, that will be used, and the cached image only needs to be generated once. Your use case is what the precacher is for. What happens for the user who isn't the sole provider of his site's content? What if someone who is not technologically savvy is meant to upload images? His images will not appear until they are explicitly precached. This is not an optimal use case. For your usage, on-the-fly caching IS failing gracefully. It's not that the content doesn't exist, it's that it hasn't been generated in the correct form yet.

Quote:they would then somehow be massively turned off by a very expected and appropriate error, the same error response that most every web server in the world would give when content does not exist
Your idea that throwing an error in this case seems strange to me. Very few sites serve solely static content these days. The majority of it is dynamically-generated. And yes, it is extremely off-putting to a user when a piece of software that is meant to display images does not display an image.

In the end, if you get a solution working, feel free to post it and we can discuss specifics at that point. Until then, we are just arguing in circles. Everything said above is my own personal opinion and the devs may disagree. I will be interested to see what comes of this.

Quote:IF that was really true you would be the worst programmers in the world. But it is not true, as I knew, and as I have now proven.
Please refrain from ad hominem attacks. Let's keep the discussion civil. Thanks.

I have never said it is impossible to change Zenphoto's architecture. What I'm saying is that, since Zenphoto's inception, this has been a distinguishing feature compared to other gallery software and our paradigm seems to work quite well for many users. If a solution to your problem is implemented, it will need to cater to both sides of the fence.

Personally, I feel that the ideal solution can be reached by reducing the size of your uploaded photos and/or creating a sequenced precacher. Yet again, just my opinion.

As one last aside, the mechanism we're talking about involves processes, not threads. They are quite different.

First there was no personal "ahem.. ad hominem" attack. Read it again, and look up the definition of the word "if".

"I have never said it is impossible to change Zenphoto's architecture. What I'm saying is that, since Zenphoto's inception, this has been a distinguishing feature compared to other gallery software and our paradigm seems to work quite well for many users."

really? here's your last quote about it:

"It is impossible to turn off on-the-fly caching without either major architecture change to Zenphoto or implementing some manner of IPC. "

(ok not ACTUALLY impossible, but are we playing word games here or what?... what we know is it is possible WITHOUT changing any architecture.)

I turned it off with one character of coding and got exactly the same behavior as other systems that don't have this "distinguishing feature" without any architecture change or IPC. One character of change (plus the option and conditional) can give your users the choice to use a traditional style server (with its traditional errors IF AND ONLY IF THE ADMIN MESSES UP) or your on-the-fly server.

It's a great distinguishing feature but also one that is easily turned off. Providing the option would give people who like everything else, but not this, to still use your code. I get why the feature is nice, but having the mentality that people either enjoy your code the way you think they should, or don't use it is kind of silly. I get that too if we're talking about a bunch of work, but we're not. We're just talking about a single conditional with a null action that gives people choice and expand your userbase to people with different usage ideas than your own.

"Personally, I feel that the ideal solution can be reached by reducing the size of your uploaded photos and/or creating a sequenced precacher. Yet again, just my opinion."

Giving a solution to a need to by suggesting one changes their need is disingenuous, especially when the real solution is simple.

Yes, a sequenced precacher is the point, but we go in circles on this: sbillard says why should he sequence the precaher when I should reduce my threads to 2. In the same post he says, reducing threads on a server is bad ... so go explain that. Discussing something with you guys is like playing basketball with a moving net. If you're going to make a sequenced precacher you don't also want the ability for one browser request to bypass that.

Content creation takes TWENTY TIMES more resources than content serving (as you now seem to get) and they CAN BE OPTIONALLY SEPARATE NEEDS. Forcing them to be linked when they don't need to be is odd.

Most recently (in the ticket) sbillard says he does think this is possible, which also goes against what you are saying (this time) and what I've proven with an extremely simple code change.

What could your massive architecture and IPC code overahaul that is just way too complicated for my small brain to get possibly change to make a static content option behave better than the one I made with one character? How could all that supposed required work to do this do better than giving a broken image icon?

Ok.. I'll give an actual answer(even though one is not needed). If you really want to make the option cool you COULD make it remove the missing images from the pages. That is NOT a massive architecture change either. Somewhere all themes plugins, pages determine what images are available. One simply needs to improve that decision to add "has all needed resized images" as oposed to just "source file exists" to the definition of "available" when creating the list. I would say that improvement is something you could do before you take the "beta function" qualifier off the option check box, but it's nothing like a requirement to roll out the option.

Other static content servers including ones far more popular than your dynamic one, just give broken icons though, and many people are very happy with that. It's true using their usual precache(import) tools that error won't happen, but again, you can also get to that as a future enhancement without a massive overhaul and without requiring use of an import tool.

You have the ability to go two or three better than them. You can give the default option for dynamic creation AND you can have an option for their static method but easier (because you can just copy the images in before hitting the precacher). So far this is all with almost zero work... AND if you want you can do a little more work and get rid of the broken image icons (in case the admin didn't do his job or something gets deleted).

(edit: above.. sbillard does NOT think it is possible)

And hey, I'm not facebook guy. It's your code. You don't owe me anything. But if you're willing to discuss it, I'm going to point out where your points don't make any sense. And while you don't owe anyone anything.. it's just a shame to let your ideals get in the way of spreading the usefulness of your work and optionally reducing resource requirements by about TWENTY TIMES over such a simple option. It's weird to have so many thousands of lines of amazing code and to pin your pride in the whole thing on a fetish with one small feature coded in relatively few lines.

I actually think we are discussing this and quite lenghtly and no one compared you with the "facebook guy" at all. I am not familar enough with the image processor code, especially not the Imagick part, to really join in the technical part.

But generally the caching is an quite integral part of the image processor and Zenphoto's feature
that all from sized to cropped ones is created automatically is quite a good feature. Especially for the "normal website owners" I do setup sites for.

It is the behaviour of Zenphoto since its creation actually and we don't get that much complains about it at all. Which is always a guide about this core behaviour.

We/I do appreciate your ideas and discussing them. It would be sbillard's decision since he would be the one who would do these changes.

Thanks. I know you didn't compare me to facebook guy. I just thought it was worth really saying that my sentiments don't align with that type of extreme thinking.

I also understand the affection with this good feature. I think I've also shown though by actual trial, that while the feature is central to the thinking, the nice structured coding used does in fact make it a piece of cake to safely remove, even if nobody ever thought to make it so easy. Clearly it was a core idea and I think even the programmers just assumed that it is inseparable since everything was built with this idea in mind and nobody ever put any thought into how to remove it. It's weird to me that sbillard still thinks it's impossible to remove after I did it though.

I might make a brief (unlike me) new summary thread explaining my findings in a shorter way. Maybe someone who does know php will carry it on some day. I might still try to hack the other half of it (the serialization). But I won't be programming the option buttons.

We surely appreciate the effort and surely also gladly look at any kind of contribution.

So I just figured out how to ask the server to generate any arbitrary size image I want with a url request (not to imply it's some clever trick.. I know it's meant to be possible). RAM usage depends on the size you resize to, not just from. This means unless you've got per-client server limits I can DOS your demo for as long as it takes to run out of sizes and crops. If you don't have at least decent total process limits, I can even swap it or crash it, with a three line bash script. I can also probably fill up a bunch of your disk space. This is not a threat. Kind of off topic, but kind of related too.

Your comment came off as a very backhanded compliment. My apologies if that was not the intention.

We're not saying that what you're asking can't be implemented, we're saying that it won't be by us. If you come up with a solution, we will take a look at it then, but do be aware that it's not guaranteed to be included in the core code either.

Suggesting that someone does not upload huge images is our typical advice, so much so that we have an article dedicated to such issues: http://www.zenphoto.org/news/why-are-some-of-my-thumbnails-sized-images-not-generated-why-does-zenphoto-make-my-server-crash-while-processing-images-

Your need can still be fulfilled by uploading smaller images and offering the full-sized versions outside of Zenphoto. Content creation would be automatic and the full-sized images would still be available.

Or the precacher should do its thing. Images that are not cached (or fail to cache) could be set as unpublished. Images that are newly uploaded could start as unpublished and the new precacher could set them to published as they are successfully cached. There will need to be warnings when caching an image fails, though, otherwise users will become confused when their image is missing without warning, thinking that the precacher worked correctly. Some ideas for your implementation.

But I still personally feel it is a better option to use the precacher and fall back on on-the-fly loading. It's the best of both worlds and, IMO, a server should be appropriately equipped to handle such a situation.

One other note is that "has all needed image sizes" is not always a static requirement for all users. It can change over time and code needs to account for that.

"Suggesting that someone does not upload huge images is our typical advice, so much so that we have an article dedicated to such issues:"

Right, so much so there are clearly many people who's original preference is to use larger images and who would benefit by some solution like this.

I saw your referenced i.php lines before you edited that out.
That seems useful. It's not set by default, it's not in the "security" tab and there's a warning on the option making it so one would be inclined not to set it. I don't quite understand the warning about memory usage yet though. I bet many public zp servers are vulnerable to this problem to some extent. Your demo is.

For serializing I'm thinking now along the lines of waiting to get a semaphor or mutex lock in i.php. That would allow the on-the-fly cacher to still work. I don't know how that plays with timeout issues, but not worse than jamming up the system with too many processes. OK, so technically, that's IPC.

well this makes things a thousand times easier,
some commenter on the php manual page:

"On the Apache webserver, ... any semaphores got and acquired by a script and not released and removed will still be automatically cleaned up by the PHP interpreter each time the script terminates."

but unless there's slightly more to it, it might still not cleanup after kill -9 or crashes and if other servers don't clean up that's a problem. Of course this is the pain with locks.

and this is bad, from the php manual:
"Support for this functions are not enabled by default. To enable System V semaphore support compile PHP with the option --enable-sysvsem ."

of course I have no idea how or if this works on windows etc.

I deleted the i.php lines because they did not do what I thought they did. That option would not prevent flooding requests to i.php. I'm not sure if this is an issue for GD, but for Imagick, MAGICK_THROTTLE could counter this.

System V semaphores are not available on Windows. For PHP, you will be far better off using a file lock. Check out fopen with the x option. Keep in mind that, as you said, you will have to implement cleaning up stale locks in case the process is killed. (This is actually fairly trivial if you write the PID to the lock file.)

trivial unless you want control of number of allowed processes, then maybe slightly harder, but still probably not too bad.
Is the x option atomic? The manual doesn't exactly say it is.

Yes. The underlying O_EXCL flag to the C lib guarantees atomicity.

Thanks. I'm not really impressed with the flooding solution. It seems like it will not reduce the number of httpd processes that one user can sit on at a time, maybe won't help memory much (I'll try and see), they can still use cpu to the max you ever want it used for image creation at least, but for much longer, and can flood out precaching requests from others who are also governed by the same communal (from the sounds of it) limit, and they can still fill up hard drive space nearly forever until other precache requests cannot be fulfilled for lack of space.

I think it's a serious issue but comes with the territory of allowing large user demand-able resource allocation I guess. I should start a new thread but I'm not sure I should have mentioned in a thread at all. I think the right solution is to have an option to dissallow all external access to i.php, and that should be the default. When the option is set it should also be true that no themes etc pass through limitlessly arbitrary user requests to the internal interface.

Of course if the user is not allowed to make arbitrary requests, then the list of requests they can make is predetermined (even if in a complex way) and precachable. There could also be higher-permission users (bloggers or whatever) who are allowed to make arbitrary requests though.

You actually had me mostly convinced that the secure image processor option should work. I thought the fix might just be to make it more default/noticable/less-discouraged. I think it's intent was to protect content though, not system resources, but I still thought it would work.

Wow.. this works awesomely .. and with about 10 lines of code, probably 5 that actually do anyting. Once the processes aren't piling up on each other, it even works pretty fast. And it could be used with OTF either enabled or disabled, but I left it enabled. Too bad the devs made me wade through all this just so I could learn enough to figure out where to put 10 philosophically obvious lines of code. In less time than you all speant arguing and writing blog posts about how to not use big images, you could have just fixed this.

Of course I rewrote the whole architecture of zp in 10 lines (but I did implement IPC)

btw I didn't use fopen. It's atomic, but writing the PID into the file is not atomic and while that can be dealt with. I took an easier way, but there's an obvious better way than that or mine with slightly more work.

Now to worry about that security threat.. well this already helps it quite a bit at least.

I didn't follow the discussion in details but if you managed something useful, please share the code…:-)

I'll sumarize for now...

Only let 1 (or configurable number) instances of the image processor run at a time.

The end. (but that was the request, and it's what works)

I'll have more time again later.