A Raspberry Pi as a decent residential proxy

栏目: IT技术 · 发布时间: 4年前

内容简介:One of our projects (A popular workaround to mitigate this problem is to pay for a proxy service to scrape these websites, sadly, we weren’t able to find a reliable provider that was within our small budget.Hence, we ended up building our own residential p

One of our projects ( cazadescuentos.net ) uses web-scrapping to scan several online stores to find discounts. Lately, we started supporting some stores that seem to block requests coming from common cloud proivdes (like AWS, DigitalOcean, etc), if you are curious, the websites are BestBuy and Costco Mexico .

A popular workaround to mitigate this problem is to pay for a proxy service to scrape these websites, sadly, we weren’t able to find a reliable provider that was within our small budget.

Hence, we ended up building our own residential proxy, right now being powered by an old Raspberry Pi model B, it’s worth adding that it wasn’t as simple as we expected, specially keeping the SSH tunnel available (more on this below).

A Raspberry Pi as a decent residential proxy

Try it

If you like to jump directly to the code or to play with it, we have open sourced the simple-http-proxy .

For the time being, I feel brave enough to even let you try the proxy without running it, hoping I won’t get a DoS, I’ll very likely remove the access once someone abuses my old Pi.

This command tells my Pi to query https://wiringbits.net by sending the custom DNT: 1 header:

curl -X POST \
  -H "Content-Type: application/json" \
  -d '{"url": "https://wiringbits.net", "headers": { "DNT": "1" }}' \
  "https://cazadescuentos.net/proxy"

How it works

The approach is actually very simple:

  • The Raspberry Pi runs a simple HTTP proxy.
  • The Pi is connected to the internet on a router exclusive for it.
  • As the Pi isn’t easily accesible from the internet, it opens a SSH tunnel to the server that will connect to the proxy served by the pi.
  • Our scrapper invokes the proxy as if it was running on localhost.

A Raspberry Pi as a decent residential proxy

About security

Security considerations:

  • Don’t expose the proxy to the world or attackers will be able to interact with your home devices.
  • Ideally, expose the proxy on an isolated network, different to the one where you connect your home devices.

Pitfalls

I ended up investing more time than expected tweaking the necessary stuff to keep the proxy working reliable, the biggest problem was related the SSH tunnel.

If you see the actual project , it includes a systemd service to keep the tunnel opened with the necessary tweaks.

The tunnel command being:

  • /usr/bin/ssh -nNT -R 9999:localhost:9000 -o ConnectTimeout=10 -o ExitOnForwardFailure=yes -o ServerAliveInterval=180 ubuntu@cazadescuentos.net

What matter the most:

ExitOnForwardFailure=yes
ServerAliveInterval=180

Future

It is very likely that if the proxy traffic increases considerably, it will get banned by some websites.

A more scalable approach could be to distribute these proxy devices into different locations, which prevents the SSH tunnel trick from being reasonable.

A possible approach is to use a queue service like AWS SQS/Kafka/etc to push the requests for scrapping a website while the proxy devices could be fighting to consume the next request, if one doesn’t complete the job, another one can try.

If you think about it, you don’t need a Raspberry Pi, you can even build a very simple Android app serving the same purpose.

In any case, this is how the proxy has been running for a couple of months, and I hope it stays like this for a while.

Found an error? We will appreciate if you submit a PR .


以上就是本文的全部内容,希望本文的内容对大家的学习或者工作能带来一定的帮助,也希望大家多多支持 码农网

查看所有标签

猜你喜欢:

本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们

产品的视角:从热闹到门道

产品的视角:从热闹到门道

后显慧 / 机械工业出版社 / 2016-1-1 / 69.00

本书在创造性的提出互联网产品定义的基础上,为读者提供了一个从0基础到产品操盘手的产品思维培养方法! 全书以互联网产品定义为基础,提出了产品思维学习的RAC模型,通过认识产品、还原产品和创造产品三个阶段去培养产品思维和产品认知。 通过大量的图片和视觉引导的方法,作者像零基础的用户深入浅出的描绘了一条产品经理的自我修养路径,并且提供了知识地图(knowledge map)和阅读雷达等工具,......一起来看看 《产品的视角:从热闹到门道》 这本书的介绍吧!

HTML 压缩/解压工具
HTML 压缩/解压工具

在线压缩/解压 HTML 代码

XML 在线格式化
XML 在线格式化

在线 XML 格式化压缩工具

HSV CMYK 转换工具
HSV CMYK 转换工具

HSV CMYK互换工具