内容简介:One of our projects (A popular workaround to mitigate this problem is to pay for a proxy service to scrape these websites, sadly, we weren’t able to find a reliable provider that was within our small budget.Hence, we ended up building our own residential p
One of our projects ( cazadescuentos.net ) uses web-scrapping to scan several online stores to find discounts. Lately, we started supporting some stores that seem to block requests coming from common cloud proivdes (like AWS, DigitalOcean, etc), if you are curious, the websites are BestBuy and Costco Mexico .
A popular workaround to mitigate this problem is to pay for a proxy service to scrape these websites, sadly, we weren’t able to find a reliable provider that was within our small budget.
Hence, we ended up building our own residential proxy, right now being powered by an old Raspberry Pi model B, it’s worth adding that it wasn’t as simple as we expected, specially keeping the SSH tunnel available (more on this below).
Try it
If you like to jump directly to the code or to play with it, we have open sourced the simple-http-proxy .
For the time being, I feel brave enough to even let you try the proxy without running it, hoping I won’t get a DoS, I’ll very likely remove the access once someone abuses my old Pi.
This command tells my Pi to query https://wiringbits.net
by sending the custom DNT: 1
header:
curl -X POST \ -H "Content-Type: application/json" \ -d '{"url": "https://wiringbits.net", "headers": { "DNT": "1" }}' \ "https://cazadescuentos.net/proxy"
How it works
The approach is actually very simple:
- The Raspberry Pi runs a simple HTTP proxy.
- The Pi is connected to the internet on a router exclusive for it.
- As the Pi isn’t easily accesible from the internet, it opens a SSH tunnel to the server that will connect to the proxy served by the pi.
- Our scrapper invokes the proxy as if it was running on localhost.
About security
Security considerations:
- Don’t expose the proxy to the world or attackers will be able to interact with your home devices.
- Ideally, expose the proxy on an isolated network, different to the one where you connect your home devices.
Pitfalls
I ended up investing more time than expected tweaking the necessary stuff to keep the proxy working reliable, the biggest problem was related the SSH tunnel.
If you see the actual project , it includes a systemd service to keep the tunnel opened with the necessary tweaks.
The tunnel command being:
-
/usr/bin/ssh -nNT -R 9999:localhost:9000 -o ConnectTimeout=10 -o ExitOnForwardFailure=yes -o ServerAliveInterval=180 ubuntu@cazadescuentos.net
What matter the most:
ExitOnForwardFailure=yes ServerAliveInterval=180
Future
It is very likely that if the proxy traffic increases considerably, it will get banned by some websites.
A more scalable approach could be to distribute these proxy devices into different locations, which prevents the SSH tunnel trick from being reasonable.
A possible approach is to use a queue service like AWS SQS/Kafka/etc to push the requests for scrapping a website while the proxy devices could be fighting to consume the next request, if one doesn’t complete the job, another one can try.
If you think about it, you don’t need a Raspberry Pi, you can even build a very simple Android app serving the same purpose.
In any case, this is how the proxy has been running for a couple of months, and I hope it stays like this for a while.
Found an error? We will appreciate if you submit a PR .
以上就是本文的全部内容,希望本文的内容对大家的学习或者工作能带来一定的帮助,也希望大家多多支持 码农网
猜你喜欢:本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们。
产品的视角:从热闹到门道
后显慧 / 机械工业出版社 / 2016-1-1 / 69.00
本书在创造性的提出互联网产品定义的基础上,为读者提供了一个从0基础到产品操盘手的产品思维培养方法! 全书以互联网产品定义为基础,提出了产品思维学习的RAC模型,通过认识产品、还原产品和创造产品三个阶段去培养产品思维和产品认知。 通过大量的图片和视觉引导的方法,作者像零基础的用户深入浅出的描绘了一条产品经理的自我修养路径,并且提供了知识地图(knowledge map)和阅读雷达等工具,......一起来看看 《产品的视角:从热闹到门道》 这本书的介绍吧!