Add a new service Houdini #20
Conversation
|
waits for the wonderful @manez avatar |
Houdini/README.md
Outdated
| - `$ cd /path/to/Houdini` and run `$ composer install` | ||
| - Then either | ||
| - For production, configure your web server appropriately (e.g. add a VirtualHost for Houdini in Apache) OR | ||
| - For development, run the PHP built-in webserver `$ php -S localhost:8888 -t src` from Hypercube root. |
| protected $log; | ||
|
|
||
| /** | ||
| * HypercubeController constructor. |
jonathangreen
force-pushed
the
issue-577-2
branch
from
40339f1 to
858870e
Compare
|
Y'all don't like to make this easy, do you?
|
jonathangreen
force-pushed
the
issue-577-2
branch
2 times, most recently
from
0541634 to
bae35d9
Compare
Add a new service for transforming images using imagemagick.
jonathangreen
force-pushed
the
issue-577-2
branch
from
bae35d9 to
7f95550
Compare
Codecov Report
@@ Coverage Diff @@
## master #20 +/- ##
=============================================
+ Coverage 63.33% 78.06% +14.73%
- Complexity 30 42 +12
=============================================
Files 3 4 +1
Lines 90 155 +65
=============================================
+ Hits 57 121 +64
- Misses 33 34 +1
Continue to review full report at Codecov.
|
|
This might be me and my setup but Also, Postman on a Mac seems to have trouble with either odd ports or PHP servers. Cause I couldn't connect at all to the PHP server. But curl worked for identify. Okay, figured out Postman. Need to specify 127.0.0.1 instead of localhost. I upped my time out to 2 minutes, but it still times out. Here is the contents of the houdini.log for my failed request...am I doing something wrong?
|
|
Looks like everything is working okay to me! Is it running on a really big file. I'm thinking the 2 minute timeout may not be enough. If you try running: Maybe just set your PHP timeout to 0, so it runs forever. I'm pretty sure we are going to have to set the timeout for these micro services to a fairly large value if not unlimited. |
|
@whikloj yeah, I ran into the same problem testing the first Hypercube PR, and unfortunately @dannylamb and my conversation in the PR is gone. But! https://github.com/Islandora-CLAW/Crayfish/tree/master/Hypercube#configuration should help out. |
|
So actually it was a JPEG, with
Would it being a JPEG to start with make a difference? |
|
🤣
For my Fedora, infinity was not long enough |
|
@whikloj I had problems with |
|
Ohhh wait, I was running this outside of my vagrant setup. Would this need to be inside vagrant? If it can do the identify command then it should work for convert...right? |
|
I'm going to try this with Apache and see if PHP -S is causing my problems. |
|
Just tried Houdini locally (not in Vagrant, but pointed to my Fedora in Vagrant) and converted a 28 MB tiff to a JP2 no prawb. Let me know how the apache setup goes (if you can at least report back what it took, we could try and convert that into some install code for claw_vagrant). |
|
@whikloj some information on what it took to get hypercube setup with apache here: Also for the infinite timeout this might help. |
|
Ok with apache outside of vagrant on my Mac and @jonathangreen's Authorization change tips (thank you for that). I still can't get it done. It takes 15 seconds to curl download the Tiff from Fedora to outside my vagrant
and really no time to convert it
I even tried pointing directly to my convert using the
Current Speed = 0, and checking top I see
So it's not calling convert correctly or something. But I trust you all and once it is incorporated in to vagrant then I'm sure it will work. |
|
@whikloj anything interesting in the Houdini log for that last one? |
|
@jonathangreen not really 😞
|
|
Interestingly enough, if I try to convert to jpeg instead of JP2, I can replicate @whikloj's issue exactly. From the console of my PHP built-in instance: From my log: |
|
It kind of looks like the process for convert is never exiting... From the timeout before it timeout before it timed out here: and from the log its calling convert like this: Just to see if its something with pipes and your version of convert can you try something like this: When I was testing I was able to convert to a jpeg okay, so I'm not exactly sure what is going on, I'll try to get mine time time out as well though so I can debug. I wonder if its something with running Houdini on a different server from Fedora. All my testing was done in the Vagrant VM. |
|
Worked fine. I got some mumbo jumbo from convert, but I viewed the resulting jpeg without issue. daniel@daniel-Latitude-3560:~/Code/PHP/Crayfish/Houdini$ cat ~/Downloads/Newspaper\ page.tiff | convert - -resize 50% jpeg:- > test.jpg
convert: Unknown field with tag 41728 (0xa300) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/881.
convert: Unknown field with tag 42016 (0xa420) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/881.
daniel@daniel-Latitude-3560:~/Code/PHP/Crayfish/Houdini$ |
|
Just in case its an imagemagick version issue can ya'll post the version of imagemagick that you have installed on the box you are testing Houdini on? Heres the version I was testing with: $ convert -version
Version: ImageMagick 6.8.9-9 Q16 x86_64 2017-02-01 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2014 ImageMagick Studio LLC
Features: DPC Modules OpenMP
Delegates: bzlib cairo djvu fftw fontconfig freetype jbig jng jp2 jpeg lcms lqr ltdl lzma openexr pangocairo png rsvg tiff webp wmf x xml zlib |
daniel@daniel-Latitude-3560:~/Code/PHP/Crayfish/Houdini$ convert -version
Version: ImageMagick 6.8.9-9 Q16 x86_64 2017-03-14 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2014 ImageMagick Studio LLC
Features: DPC Modules OpenMP
Delegates: bzlib cairo djvu fftw fontconfig freetype jbig jng jpeg lcms lqr ltdl lzma openexr pangocairo png rsvg tiff wmf x xml zlibLooks like I have a slightly newer version. Wouldn't be too concerned. @whikloj? |
|
So here is the info, I'm guessing it is a shell thing. |
|
Side question...and not for right now but have we considered http://php.net/manual/en/book.imagick.php? I have never used it and I don't know that it won't be super slow or error prone. Just wondering. |
|
Hrm... Not to complicate things further, but I set things up on the Vagrant environment to make sure it's apples to apples and I'm still getting that strange issue. Admittedly, the test tiff I'm using comes from University of Manitoba... so... Maybe @jonathangreen can share the tiff he's using and we can make sure there's nothing funky going on there. I also played with how the streams get closed in Crayfish Commons, and didn't close STDIN until after the process finished. That actually induced the same error when trying to run identify, so I don't think that's the problem, but maybe there is something with how we're playing with streams. |
|
My tiff was not very sophisticated. I needed something with text to test hyper cube against to I took a screenshot of a GitHub ticket and converted it to TIFF to see what Hypercube would so with it. Then since it was already in Fedora thats what I was using to shrink and rotate etc with Houdini. Maybe the key is that is was fairly small. I attached it here for what its worth. I just did a destroy and I'm brining up the vagrant again, see something about my environment was weird and I see the same issue now. Added it here: infile.zip |
|
so looks like claw vagrant is having some issues maybe? I give up for tonight. I have some other business to attend to tomorrow, but I'll see if I can get an environment together where this fails and find a solution to it in the next couple days. If anyone has any ideas whats up though, they are more then welcome 😄 |
|
Verified explosion: https://gist.github.com/ruebot/0be99a86e332ae15ab7f679ee2e9adbf Did this while doing a clean test build for islandora-deprecated/claw_vagrant#33 I'll dig into it more tomorrow morning. |
|
So I did some more digging and I am positive that the streamed data is not getting to convert, so I hang in this loop. But I solved it and I'm not sure I understand how. I'm not very fluent in streams but...I stopped waiting. I commented out the above loop which leaves And I got the image back. |
|
The vagrant build issue is claw_jsonld: https://gist.github.com/ruebot/0be99a86e332ae15ab7f679ee2e9adbf#file-gistfile1-txt-L7-L10 |
|
Yeah, so I was messing around last night too and it seems that PHP will hang if either output buffer with proc_open gets larger than 4 kB. So we gotta read from STDOUT to a temp stream during that loop. I have a sort of working fix for that. I can issue a PR to Crayfish Commons later today. |
|
Maybe we want to look into using a library to manage the streams, something like: Might make our life easier or might be a blackhole I dunno... |
|
claw_vagrant build resolved with islandora-deprecated/drupal-project@0c7090a |
|
@jonathangreen Thank looks interesting, keeping in mind that I'm not well versed in stream issues. Have you used that before? |
|
Or what about https://symfony.com/doc/current/components/process.html ?? |
|
@whikloj that is a fancy wrapper around http://php.net/manual/en/function.pcntl-fork.php ? I have used that more brute force child with success. But if symfony does the process handling for you that sounds like a good choice. reactphp is cool also, but requires a different approach and integration. Mainly, if you go reactphp, all works better if the whole thing is reactphp |
|
While reading the reactphp thing I also noticed this warning. It is not the issue I was running against, but we might want to be aware of it in case we get one in the wild. ...you know eventually...in the wild... 🐯 |
|
Y'all should try this out: Islandora/Crayfish-Commons#4 If this blows up again, I think it'd be a good idea to go down the https://github.com/reactphp/stream rabbit hole. EDIT: Or any of the other libraries mentioned. I should refresh my cache more often. |
|
With @dannylamb's change to Crayfish-Commons (Islandora/Crayfish-Commons#4) I am now 👍 to this service. |
|
@whikloj I'll give you the merge honour |
6 participants
GitHub Issue
Resolves:
Islandora/documentation#577
What does this Pull Request do?
Add a new service for transforming images using imagemagick.
How should this be tested?
http://localhost:8080/fcrepo/rest/some/crazy/pathphp -S localhost:8088 -t /path/to/Houdini/srchttp://localhost:8088/identify/some/crazy/pathhttp://localhost:8088/convert/some/crazy/pathInterested parties
@dannylamb