Problems with UALG's proxy (of the SRTM script)

In BGU the proxy accepts FTP connections and everything runs ok (as long the Iptables is off and the enviromment has the proxy's url). The problem is that the proxy from UALG only accepts HTTP, I have tried to upgrade the FTP connection of the script to deal with it, but it was a nightmare.

The simplest solution was to use the wget command instead of an FTP connection

The the case of running from BGU or UALG the enviromments need to have the proxy:
BGU: http://wwwbgu.ac.il:8080
UALG: http://proxy1.si.ualg.pt:8080

#!/usr/bin/perl

use strict;
use warnings;
use Net::FTP;
#variable declaration
#specific varibles, change here according to needs
my $ftp_url="ftplmu.jrc.it";
my $ftp_pwd="/pub/SRTM_v3/SRTM_Data_GeoTiff";
my $ftp_file;
my $cmd;
my $ftp;
#input variables
my $max_x;
my $min_x;
my $max_y;
my $min_y;
#other variables
my $temp_min_y;
my $formated_number_y;
my $formated_number_x;
my $file_url;
#user input
print "Enter tile number of maximum long. (x): ";
$max_x=<STDIN>;
chomp($max_x);
print "Enter tile number of minimum long. (x): ";
$min_x=<STDIN>;
chomp($min_x);
print "Enter tile number of maximum lat. (y): ";
$max_y=<STDIN>;
chomp($max_y);
print "Enter tile number of minimum lat. (y): ";
$min_y=<STDIN>;
chomp($min_y);
#for loop to generate the url and open the FTP connection
$temp_min_y=$min_y;
for ( ;$min_x<=$max_x; $min_x++) { #some nice C/C++ sytanx $formated_number_x=sprintf("%02d",$min_x); $min_y=$temp_min_y; for( ;$min_y<=$max_y; $min_y++) { $formated_number_y=sprintf("%02d",$min_y); $ftp_file="srtm_".$formated_number_x."_".$formated_number_y.".zip"; #using wget to get the file, wget will use the system's variable of proxy $cmd="wget ftp://".$ftp_url.$ftp_pwd."/".$ftp_file; system($cmd); } }

Downloading the SRTM V3.0 using a PERL script

GeoTiff is the best format to download because it runs very well with GDAL and GRASS, the tiles are in 5º by 5º. I am interrested in area between 35ºE -> 100º E; 30ºN -> 60ºN, this is a huge square from the western med. till the middle of China. The main objective is only the Aral Sea, but since I need to catch all the NIS countries the area has to be relatively big (Kazakhstan is HUGE!!!)

From the map I need all the tiles from x=44 y=6 till x=56 y=1 (left bottom till rigth top) this gives a big amount of clicks/downloads !!!! Using the approach of lets spend 99% of the time making a script that will do 100% of the job in 1% of the time, I decided to make a simple PERL scrip.

I am no PERL expert, I normally program in C/C++ so I am not familiar with PERL, but it was relatively simple because I had a book around the house called "Perl for C programmers" and the code lines for the FTP access where more or less copied from the explanation of how the Net::FTP module works.

The tile files have a structure like srtm_X_Y.zip so the script asks for the minimal and maximal number for x and y, after it makes the URL string and opens an FTP connection to the italian server. The script only has the debug dumping of the FTP connection.

Just copy/paste the next files to a file and run it:

#!/usr/bin/perl
#The script generates the URL of the files to download
#then it opens an FTP connection a gets the file
#simple, clean without much programming advances and it works
#note: the file code is something like this srtm_X_Y.zip
use strict;
use warnings;
use Net::FTP;
#variable declaration
#specific varibles, change here according to needs
my $ftp_url="ftplmu.jrc.it";
my $ftp_pwd="/pub/SRTM_v3/SRTM_Data_GeoTiff";
my $ftp_file;
my $ftp;
#input variables
my $max_x;
my $min_x;
my $max_y;
my $min_y;
#other variables
my $temp_min_y;
my $formated_number_y;
my $formated_number_x;
my $file_url;
#user input
print "Enter tile number of maximum long. (x): ";
$max_x=<STDIN>;
chomp($max_x);
print "Enter tile number of minimum long. (x): ";
$min_x=<STDIN>;
chomp($min_x);
print "Enter tile number of maximum lat. (y): ";
$max_y=<STDIN>;
chomp($max_y);
print "Enter tile number of minimum lat. (y): ";
$min_y=<STDIN>;
chomp($min_y);
#for loop to generate the url and open the FTP connection
$temp_min_y=$min_y;
for ( ;$min_x<=$max_x; $min_x++) { #some nice C/C++ sytanx $formated_number_x=sprintf("%02d",$min_x); $min_y=$temp_min_y; for( ;$min_y<=$max_y; $min_y++) { $formated_number_y=sprintf("%02d",$min_y); $ftp_file="srtm_".$formated_number_x."_".$formated_number_y.".zip"; #creating the FTP object and getting the file $ftp = Net::FTP->new($ftp_url, Debug=>1, Passive=>1) or die "It is not connecting: $@";
$ftp->login("anonymous");
$ftp->cwd($ftp_pwd);
$ftp->message;
$ftp->binary;
$ftp->get($ftp_file) or die "Get failed ", $ftp->message;
$ftp->quit;
}
}

Getting SRTM data (version 3)

The SRTM data is the altimetric data created by the Shuttle Radar Topography Mission, and it is available in a resolution of 3-arc second (90meter) outside the USA. Version 3.0 has finshed SRTM data, DEM void's filled, clips of the coastlines.

Methodology of SRTM V3.0

WebGIS to download tiles of SRTM V3.0

This server has the option of mirror picking and a direct access to the FTP

I tested the 3 mirrors for velocity of download the srtm_48_05.zip file, my location is Portugal and my ISP says that my ADSL is at 24mbits (Ya shore......) and the test was at 4:54PM (Lisbon Time) in 29 Nov. 2006
HTTP connection of CSI server: 268kB/sec
FTP connection of CSI server: 154kB/sec

HTTP connection of JRC(IT) server: 254kB/sec
FTP connection of JRC(IT) server: 470kB/sec

HTTP connection of Kings College (UK) server: 118 kB/sec
FTP connection of Kings College (UK) server: 351 kB/sec

So the velocity doesn't change much but it seems (at least for me) that the Italian server is better for my location

Also there are some FTP servers with version 2.0 and 1.0:
ftp for version 1.0 and 2.0