Neh – Execute any script or program from Nginx location directives

栏目: IT技术 · 发布时间: 4年前

内容简介:I had a simple question: can I run a bash script straight from Nginx?Everyone encounters a moment in which they’re held back by a limitation of a certain thing they use. Whether that’s your morning citrus juicer that doesn’t seem to catchWe encounter all s

I had a simple question: can I run a bash script straight from Nginx?
Finding no answer good enough for my use case, I decided to build my own solution: Neh .

Everyone encounters a moment in which they’re held back by a limitation of a certain thing they use. Whether that’s your morning citrus juicer that doesn’t seem to catch all the pits in your wake-up juice.

We encounter all sorts of limitations in our day-to-day life. It doesn’t always have to be a frustation that causes you to solve these limitations. In my case – although unrelatable – it’s the limitation of not being able to run a bash script from an nginx location directive . For the purposes of this article, I’m assuming you know what all of those things are. If you don’t know what any of this means, don’t worry, but the rest of the article is probably not gonna make a lot of sense. (Not all the articles of this blog are for everyone I suppose )

What was the problem again?

When you use nginx, you usually use it as either a reverse proxy or a simple web hosting server. Most people don’t use it much outside of these purposes. Whether you run larger services, you are into devOps or you find it fun to poke around with stuff like this (I’m mostly the latter two), you can find yourself using a lot of other features too.

One of the features that I was looking for was a simple way to execute a bash script from one of the location directives of an nginx configuration. Usually I write a small service that responds through a reverse proxy. Although that was fun in the beginning, I’m starting to notice that my use cases are usually too small for writing a service. The truth is: I don’t enjoy making a side project out of these small problems. I found that writing a simple bash script is the best way to go about these smaller problems. I already do this on my computer by having a big ~/.bin folder with all sorts of scripts. The problem remains: how do you solve this on the internet when it comes to API’s and services? Well, you look for a way to execute a bash script through nginx.

My search through “the internets”

Okay, so I have to be honest here. I was gonna write this part of the article showing how hard it was to find a solution. After making Neh and deciding to write the article, it seems that my first search query finds a good answer .

When I searched the first time, most of the answers that I found revolved around using things like “FastCGI with PHP” . I wasn’t gonna use PHP to call a bash script, that would be overdoing it probably. I also found a dead forum post on nginx.org with a poor citizen of the internet with the same question as I had.

Okay, so luckely for me, I ended up at a StackOverflow thread that had the answer to my itching question.

location /my-website {
  content_by_lua_block {
    os.execute("/bin/myShellScript.sh")
  } 
}

Ofcourse! Use the HTTPLuaModule from nginx! This is the answer to all my problems. I also wanted to know the output of the script, so my config ended up looking like this:

location /my-website {
  content_by_lua_block {
    local file = assert(io.popen("/bin/myShellScript.sh"))
    local result = assert(file:read('*all'))
    ngx.say(result)
    file:close()
  }
}

So yay! I can now execute my shell script straight from an nginx location directive!

But I couldn’t just leave it there. There was one particular application I wanted to use it for: GitHub webhooks . Particularly, I wanted it for the blog that you’re reading right now. This website runs on Hugo and needs to be compiled everytime there is a push. The code above is not flexible enough to support that use case.

So, I made sure that it was.

Introducing Neh

Neh is a small lua script that provides the perfect framework for your one-off scripts or even programs through nginx. It has the features of the above code and some extras like:

  • Passing all of the request headers as Environment Variables.

    So User-Agent becomes a readable USER_AGENT variable

  • Send any data sent through the request as data through stdin !

    Have a reliable cross-language/cross-program way to receive the data.

  • Sending your stdout as a response, chunked instead of a big response buffer.

    It makes it easier to stream larger responses like files.

  • Being able to write to fd #3 to write headers of the response.

    You can easily manipulate the headers through a file descriptor by writing to it!

  • A file descriptor (#4) for sending commands to Neh.

    It makes Neh able to do actions on your behalf on the lower level.

The last feature is useful for things like ending the connection but continuing the script. All these features proved useful with GitHub webhooks, because they need a way to read the request headers, read the request JSON and send back a reponse immediately after.

Installing Neh can be done with this simple one-liner:

curl https://raw.githubusercontent.com/oap-bram/neh/master/install.sh | sh

Setting up Neh is easy! Just set a nginx variable in your location directive and let the content_by_lua_file point to Neh! The rest is done for you!

location /hooks/github-commit {
    set $execute_file /home/bram/blog/github-commit-hook.sh; # The file I want to execute
    content_by_lua_file /usr/lib/neh/neh.lua; # Execute the request and file with neh
}

Then I created a script, for the purpose of this blog. I’ve omitted the part that actually does the git pull ing and the hugo ing. This example just reads the body, and passes it through OpenSSL to make a HMAC hash. This is to verify the request is actually from GitHub.

#!/bin/bash
# Hook that is called when the github-commit hook is run

# Before we send any data I set the Content-Type to text/plain
# This is already done automatically by Neh, but I wanted to showcase the
# feature anyway 

echo "Content-Type: text/plain" >&3

# The secret you set up on GitHub
secret="a-secret-im-obviously-not-leaking-through-a-blog-post"

# Read the content of the body from stdin
body=$(cat <&0)

# Make a hash based on the body and digest into a sha1 HMAC as given by GitHub
hash="sha1=$(echo -n "$body" | openssl dgst -sha1 -hmac "$secret" | cut -d ' ' -f2)"

# Compare the hash with the request header generated by Neh
if [[ "$hash" != "$X_HUB_SIGNATURE" ]]; then
    # Fail the request if the signatures do not come across.
    echo "Hook verification failed"
    exit -1
fi

# Respond to the webhook with a message of success!
echo "Hook verification successful!"

# Immediately end the request afterward by writing to the command file
# descriptor #4
echo "END_REQUEST" >&4

# Write the rest of the output to /dev/null because we can't write to the
# response body anymore
exec >/dev/null
exec 2>/dev/null

# Actually do the rest of the work required to update the website

...

And that’s it! I made a relatively small script for a GitHub webhook like that! Nothing stops you from doing it in Ruby, Node.js, Python or even a compiled Go binary. You can execute any program with Neh and use the environment variables and stdin/stdout in your language/platform of choice!

What I’ve learned from this project

Damn, have I learned a lot on this project. I thought setting this up would be somewhat trivial, but because of Lua’s close-to-C nature, I was quickly forced to go deep on most of the problems that I encountered.

Here are some problems that taught me a ton:

  • Pipes aren’t supported by lua nor nginx lua out of the box.

    Well, suppose I’ll just build my own support with luaposix . Sure learned how pipes work on a Unix level now

  • I need to run multiple programs, how do you even do that?

    Turns out fork(2) is the way to do this. I heard about forking before, I even used it back when I was just a script kiddie. Have I fully internalized what it does or how it works? I have now!

  • It’s fun to have a side project with a reachable goal!

    I used to have side projects that have a clear goal, but sure as hell not a reachable one. I’m glad I stumbled on one that does. It sure is more motivating to finish it!

And that’s it! You can check out the project on GitHub.

Also,subscribe to my blog through RSS if you can, or share my article on social media by using the buttons on the top left!

Photo by Victor Aznabaev


以上就是本文的全部内容,希望本文的内容对大家的学习或者工作能带来一定的帮助,也希望大家多多支持 码农网

查看所有标签

猜你喜欢:

本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们

长尾理论2.0

长尾理论2.0

安德森 / 乔江涛、石晓燕 / 中信出版社 / 2009-5 / 42.00元

《长尾理论2.0》是克里斯·安德森对所有问题最明确的回答。在此书中,他详细阐释了长尾的精华所在,揭示了长尾现象是如何从工业资本主义原动力——规模经济与范围经济——的矛盾中产生出来的。长尾现象虽然是明显的互联网现象,但其商务逻辑本身,却是从工业经济中自然而然“长”出来的,网络只是把酝酿了几十年的供应链革命的诸多要素简单地结合在一起了。同时,长尾理论转化为行动,最有力、最可操作的就是营销长尾,通过口碑......一起来看看 《长尾理论2.0》 这本书的介绍吧!

HTML 压缩/解压工具
HTML 压缩/解压工具

在线压缩/解压 HTML 代码

CSS 压缩/解压工具
CSS 压缩/解压工具

在线压缩/解压 CSS 代码

Base64 编码/解码
Base64 编码/解码

Base64 编码/解码