16 Februari 2017 - 00:09:45 - Read: 337

Bot vs AJAX. Scrapping Halaman Website Yang Di load Dengan Javascript.

Masih ingat dengan statement di artikel Grab Free Proxy Lists Dari Frontend HideMyAss Tanpa API sebelumnya?

...bahwa nyaris tidak ada interface yang tidak bisa kita grabbing, selama tidak dilindungi oleh advanced captcha, atau delay realtime load content seperti AJAX. Bisa, tapi mungkin kalo gak bikin muntah, minimal bikin kejang-kejang. :p

Nah, kali ini kita akan membahas cara meng-grab konten dihalaman website yang diload menggunakan AJAX. Dan yarp, artikel ini masih seputar grabbing list free proxy untuk aplikasi HTTP Tools.

Tools ajib semacam cURL tidak bisa melakukan hal ini karena eksekusi konten dilakukan di sisi client oleh javascript. Diluar jangkauan curl tentunya. Awalnya kepikiran buat men-delay curl sebelum meload seluruh halaman web, tapi setelah dipikir-pikir lagi, AJAX tidak dieksekusi bersamaan ketika sebuah URL diminta, melainkan AJAX sendiri melakukan HTTP Request sendiri ke sisi server. Berarti cara lainnya adalah, curl harus mengikuti kemana HTTP Request yang dilakukan oleh AJAX.

Ngoding buat mencari otomatis kemana saja HTTP Request yang dilakukan oleh AJAX dari sebuah source code adalah bunuh diri. Jadi kita bantu dengan cara manual. Bisa menggunakan Developer Console Browser, atau dalam kasus kali ini kita menggunakan plugin Firefox, Live HTTP Header. Dan target yang akan kita ambil kontennya adalah website https://hidester.com/proxylist/.

 

Tampilan konten list proxy di web hidester.com

 Tampilan list proxy diatas diload menggunakan AJAX. Sehingga jika kita grab langsung url nya, kita tidak akan menemukan konten diatas. Jadi kita buka terlebih dahulu halaman https://hidester.com/proxylist/ sambil menjalankan Live HTTP Header.

live-http-header

Bisa kita lihat salah satu HTTP Request dari halaman web hidester.com adalah ke alamat https://hidester.com/proxydata/php/data.php?mykey=data&offset=0&limit=10&orderBy=latest_check&sortOrder=DESC&country=&port=&type=undefined&anonymity=undefined&ping=undefined&gproxy=2 .

Mari kita grab...

jamz@jdeepin:~/www/curlme$ curl "https://hidester.com/proxydata/php/data.php?mykey=data&offset=0&limit=10&orderBy=latest_check&sortOrder=DESC&country=&port=&type=undefined&anonymity=undefined&ping=undefined&gproxy=2" -H "Content-Type: application/json"

Ups, kosong. Berarti mari kita pura-pura load API gratis ini dari halaman utamanya

jamz@jdeepin:~/www/curlme$ curl "https://hidester.com/proxydata/php/data.php?mykey=data&offset=0&limit=10&orderBy=latest_check&sortOrder=DESC&country=&port=&type=undefined&anonymity=undefined&ping=undefined&gproxy=2" -H "Content-Type: application/json" -e "https://hidester.com/proxylist/"

[{"IP":"223.114.76.148","PORT":9999,"latest_check":1487170229,"ping":359,"connection_delay":0,"country":"CHINA","down_speed":0,"up_speed":0,"proxiescol":null,"anonymity":"Elite","type":"http","google_proxy":0},{"IP":"223.114.77.188","PORT":9999,"latest_check":1487170229,"ping":297,"connection_delay":0,"country":"CHINA","down_speed":0,"up_speed":0,"proxiescol":null,"anonymity":"Elite","type":"http","google_proxy":0},{"IP":"223.198.53.242","PORT":8998,"latest_check":1487170229,"ping":375,"connection_delay":0,"country":"CHINA","down_speed":0,"up_speed":0,"proxiescol":null,"anonymity":"Elite","type":"http","google_proxy":0},{"IP":"223.198.6.19","PORT":8998,"latest_check":1487170229,"ping":375,"connection_delay":0,"country":"CHINA","down_speed":0,"up_speed":0,"proxiescol":null,"anonymity":"Elite","type":"http","google_proxy":0},{"IP":"223.199.27.221","PORT":8998,"latest_check":1487170229,"ping":422,"connection_delay":0,"country":"CHINA","down_speed":0,"up_speed":0,"proxiescol":null,"anonymity":"Elite","type":"http","google_proxy":0},{"IP":"223.204.11.131","PORT":8080,"latest_check":1487170229,"ping":250,"connection_delay":0,"country":"THAILAND","down_speed":0,"up_speed":0,"proxiescol":null,"anonymity":"Transparent","type":"http","google_proxy":0},{"IP":"223.207.31.124","PORT":8080,"latest_check":1487170229,"ping":250,"connection_delay":0,"country":"THAILAND","down_speed":0,"up_speed":0,"proxiescol":null,"anonymity":"Anonymous","type":"http","google_proxy":0},{"IP":"223.223.132.106","PORT":8080,"latest_check":1487170229,"ping":3219,"connection_delay":0,"country":"INDIA","down_speed":0,"up_speed":0,"proxiescol":null,"anonymity":"Elite","type":"http","google_proxy":0},{"IP":"223.245.18.230","PORT":8998,"latest_check":1487170229,"ping":266,"connection_delay":0,"country":"CHINA","down_speed":0,"up_speed":0,"proxiescol":null,"anonymity":"Elite","type":"http","google_proxy":0},{"IP":"223.246.6.50","PORT":8998,"latest_check":1487170229,"ping":281,"connection_delay":0,"country":"CHINA","down_speed":0,"up_speed":0,"proxiescol":null,"anonymity":"Elite","type":"http","google_proxy":0}]

dan... Taraaaa.... :D

Mari dipilih..dipilih...  ^_^

Yarp, semoga tulisan ini bermanfaat buat bot kalian.
Happy fecthing... :D