内容简介:I had a simple question: can I run a bash script straight from Nginx?Everyone encounters a moment in which they’re held back by a limitation of a certain thing they use. Whether that’s your morning citrus juicer that doesn’t seem to catchWe encounter all s
I had a simple question: can I run a bash script straight from Nginx?
Finding no answer good enough for my use case, I decided to build my own
solution: Neh
.
Everyone encounters a moment in which they’re held back by a limitation of a certain thing they use. Whether that’s your morning citrus juicer that doesn’t seem to catch all the pits in your wake-up juice.
We encounter all sorts of limitations in our day-to-day life. It doesn’t always have to be a frustation that causes you to solve these limitations. In my case – although unrelatable – it’s the limitation of not being able to run a bash script from an nginx location directive . For the purposes of this article, I’m assuming you know what all of those things are. If you don’t know what any of this means, don’t worry, but the rest of the article is probably not gonna make a lot of sense. (Not all the articles of this blog are for everyone I suppose )
What was the problem again?
When you use nginx, you usually use it as either a reverse proxy or a simple web hosting server. Most people don’t use it much outside of these purposes. Whether you run larger services, you are into devOps or you find it fun to poke around with stuff like this (I’m mostly the latter two), you can find yourself using a lot of other features too.
One of the features that I was looking for was a simple way to execute a bash
script from one of the location directives of an nginx configuration. Usually I
write a small service that responds through a reverse proxy. Although that was
fun in the beginning, I’m starting to notice that my use cases are usually
too small for writing a service. The truth is: I don’t enjoy making a side
project out of these small problems. I found that writing a simple bash script
is the best way to go about these smaller problems. I already do this on my computer
by having a big ~/.bin
folder with all sorts of scripts.
The problem remains: how do you solve this on the internet when it comes to
API’s and services? Well, you look for a way to execute a bash script through nginx.
My search through “the internets”
Okay, so I have to be honest here. I was gonna write this part of the article showing how hard it was to find a solution. After making Neh and deciding to write the article, it seems that my first search query finds a good answer .
When I searched the first time, most of the answers that I found revolved around using things like “FastCGI with PHP” . I wasn’t gonna use PHP to call a bash script, that would be overdoing it probably. I also found a dead forum post on nginx.org with a poor citizen of the internet with the same question as I had.
Okay, so luckely for me, I ended up at a StackOverflow thread that had the answer to my itching question.
location /my-website { content_by_lua_block { os.execute("/bin/myShellScript.sh") } }
Ofcourse! Use the HTTPLuaModule from nginx! This is the answer to all my problems. I also wanted to know the output of the script, so my config ended up looking like this:
location /my-website { content_by_lua_block { local file = assert(io.popen("/bin/myShellScript.sh")) local result = assert(file:read('*all')) ngx.say(result) file:close() } }
So yay! I can now execute my shell script straight from an nginx location directive!
But I couldn’t just leave it there. There was one particular application I wanted to use it for: GitHub webhooks . Particularly, I wanted it for the blog that you’re reading right now. This website runs on Hugo and needs to be compiled everytime there is a push. The code above is not flexible enough to support that use case.
So, I made sure that it was.
Introducing Neh
Neh is a small lua script that provides the perfect framework for your one-off scripts or even programs through nginx. It has the features of the above code and some extras like:
-
Passing all of the request headers as Environment Variables.
So
User-Agent
becomes a readableUSER_AGENT
variable -
Send any data sent through the request as data through
stdin
!Have a reliable cross-language/cross-program way to receive the data.
-
Sending your
stdout
as a response, chunked instead of a big response buffer.It makes it easier to stream larger responses like files.
-
Being able to write to fd #3 to write headers of the response.
You can easily manipulate the headers through a file descriptor by writing to it!
-
A file descriptor (#4) for sending commands to Neh.
It makes Neh able to do actions on your behalf on the lower level.
The last feature is useful for things like ending the connection but continuing the script. All these features proved useful with GitHub webhooks, because they need a way to read the request headers, read the request JSON and send back a reponse immediately after.
Installing Neh can be done with this simple one-liner:
curl https://raw.githubusercontent.com/oap-bram/neh/master/install.sh | sh
Setting up Neh is easy! Just set a nginx variable in your location directive and
let the content_by_lua_file
point to Neh! The rest is done for you!
location /hooks/github-commit { set $execute_file /home/bram/blog/github-commit-hook.sh; # The file I want to execute content_by_lua_file /usr/lib/neh/neh.lua; # Execute the request and file with neh }
Then I created a script, for the purpose of this blog. I’ve omitted the part that
actually does the git pull
ing and the hugo
ing. This example just reads the
body, and passes it through OpenSSL to make a HMAC hash.
This is to verify the request is actually from GitHub.
#!/bin/bash # Hook that is called when the github-commit hook is run # Before we send any data I set the Content-Type to text/plain # This is already done automatically by Neh, but I wanted to showcase the # feature anyway echo "Content-Type: text/plain" >&3 # The secret you set up on GitHub secret="a-secret-im-obviously-not-leaking-through-a-blog-post" # Read the content of the body from stdin body=$(cat <&0) # Make a hash based on the body and digest into a sha1 HMAC as given by GitHub hash="sha1=$(echo -n "$body" | openssl dgst -sha1 -hmac "$secret" | cut -d ' ' -f2)" # Compare the hash with the request header generated by Neh if [[ "$hash" != "$X_HUB_SIGNATURE" ]]; then # Fail the request if the signatures do not come across. echo "Hook verification failed" exit -1 fi # Respond to the webhook with a message of success! echo "Hook verification successful!" # Immediately end the request afterward by writing to the command file # descriptor #4 echo "END_REQUEST" >&4 # Write the rest of the output to /dev/null because we can't write to the # response body anymore exec >/dev/null exec 2>/dev/null # Actually do the rest of the work required to update the website ...
And that’s it! I made a relatively small script for a GitHub webhook like that! Nothing stops you from doing it in Ruby, Node.js, Python or even a compiled Go binary. You can execute any program with Neh and use the environment variables and stdin/stdout in your language/platform of choice!
What I’ve learned from this project
Damn, have I learned a lot on this project. I thought setting this up would be somewhat trivial, but because of Lua’s close-to-C nature, I was quickly forced to go deep on most of the problems that I encountered.
Here are some problems that taught me a ton:
-
Pipes aren’t supported by lua nor nginx lua out of the box.
Well, suppose I’ll just build my own support with luaposix . Sure learned how pipes work on a Unix level now
-
I need to run multiple programs, how do you even do that?
Turns out
fork(2)
is the way to do this. I heard about forking before, I even used it back when I was just a script kiddie. Have I fully internalized what it does or how it works? I have now! -
It’s fun to have a side project with a reachable goal!
I used to have side projects that have a clear goal, but sure as hell not a reachable one. I’m glad I stumbled on one that does. It sure is more motivating to finish it!
And that’s it! You can check out the project on GitHub.
Also,subscribe to my blog through RSS if you can, or share my article on social media by using the buttons on the top left!
Photo by Victor Aznabaev
以上就是本文的全部内容,希望本文的内容对大家的学习或者工作能带来一定的帮助,也希望大家多多支持 码农网
猜你喜欢:本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们。