FlashGot - a beginners guide to easy downloading with Firefox

FlashGot is the best Firefox download manager integration around. That is their words and they may very well be true. FlashGot ( http://flashgot.net/ ) is a tool to integrate the website with a download manager of your choice.

In regard to download managers then we are in business with tools like DownThemAll ( http://www.downthemall.net/ ), which is free and the commercial tool ReGet ( http://www.reget.com/en/ ). I will demonstrate both here in this micro tutorial. We will also have a quick look into some of the configuration possibilities of FlashGot.

Quote: "Spend less time downloading and more time on organizing". 

The image screenshot below shows the Preferences for FlashGot. From the red marked circles you can see that I have a possibility to choose the default download manager as either DTA (Down Them All), ReGet or the build in browser download capability. I am most familiar with ReGet as it gives me the most information and control, but the choice matters little as both tools manage to download properly.

FlashGot supports a wide range of download managers whereas the most popular probably is DownThemAll, FlashGet, GetRight, jDownloader and ReGet.

FlashGot basically has two modes, either you choose to download 1 link or you choose to download the entire page, and everything that is linked to from the page, one level deep. Let's see how things looks using DownThemAll and choosing 'FlashGot All'.

If we switch the default download manager to ReGet then we get a slightly different picture although they are quite similar. With DTA some files are checked as which to download. With ReGet all files are picked as to be downloaded. Both tools require you to hand pick which, because in this case you may want only the default size video and the HD largest quality video.

With both DTA and ReGet the next step is to click OK and let the downloading begin. If you right click a link and choose 'FlashGot Link' which is the single file download mode then you get thrown directly into the single file download mode of the download manager you have set the preference too. Again it is the OK button and your download starts.

The force of download managers like DTA and ReGet are their ability to stop and resume huge downloads with multiple threads. Downloading a 2.18Gb file may take a little while and the best way is to use a tool like DTA or ReGet. Give it a try.

Fixing the web? - downloading by forums

This summary is not available. Please click here to view the post.

Getting the daily porn fix - bittorrent

Sometimes you get stuck with nothing to download and yet the need is there. A true addiction or collectors syndrome.

Yet you know that it is out there you just need to get it. And help is on the way. See there are quite a few places which is open to your disposal if you do good.

There's pr0n on the web, usenet, irc, ftp and on torrents. So if you are up for it then you should get yourself a torrents client. And the one which puts the least pressure on your hardware is by far μTorrent. You can find μTorrent on http://www.utorrent.com. There are quite a bunch of others available and one of the popular ones is Azurerus/Vuze which is a opensource project in Java. You can find it here: http://azureus.sourceforge.net/.

So you need to setup your torrent client and the most important thing is A) use encryption and B) make sure that your torrent client can be contacted from the outside. There are excellent articles about how to proper set up your torrent client, but basically they are all about port forwarding and allowing traffic through your firewall.

Next up is a place with torrents. And the ones you should be looking for are one of the 3 greats. And they are (no specific order).

• Cheggit
• Pure T'n'A
• Empornium

There you can get pretty much everything. Google them to find the url's. You will have to create an account, you will have to let others download from you and you will have to let your box be online just a little bit more than normal. But putten that aside then you only have to learn to browse torrents and to stay online.

And just a heads up on A) DO READ THE SITE RULES and B) Do not be a complete leech.

Understanding the URL / links

URL's or links as they are often called can be quite complex. This article will dig into urls and try to explain them. Let's look at 4 url examples below.

1)hxxp://aaabbbcccd.n1.brazzers.com/members/?action=scenes&site_id=5&order_by=date&order_order=desc
2)hxxp://aaabbbcccd.n1.brazzers.com/members/?action=scene&tab=home&scene_id=2776
3)hxxp://members.bangbros.com/milfsoup/updates-1.html
4)hxxp://members.bangbros.com/membercheck?path=ms4276/streaming&fname=ms4276500k.wmv


There are several parts to an url. The easiest is no. 3 which is a plain a simple url. The url consists of 4 parts.

hxxp://www.sitename.com/path_on_server/web_page.html

Part 1
The first part is the protocol, in our case the protocol is http (hxxp). You may have experienced protocols as mms:, https:, ftp:, rtsp: or pnm:. But let us concentrate on http.

Part 2
The second part is the name on the server. A servername consists of a server and a domainname. As example: members.bangbros.com, members is the server, bangbros.com is the domain. Together they define a unique webserver.

Part 3
The second part is the path on the server, which boils down to the directory on the server. Now the directory is a virtual directory. The real physical path could very well be something like /user/bangbros/www/milfsoup/ (as example, if any Unix like server was used) or c:\inetput\www\www.bangbros.com\milfsoup\ (if Windows is used as web server). But you should not care about the real physical path.

Part 4
This is the last part of the simple url. It tells you the name of the html file on the harddisk on the server. A plain .html file is often user created, which basically means that some guy or girl has actually taken time in DreamWeaver or FrontPage (God forbid it) or some other tool and actually written and composed the page. Then he or she has saved the page to the proper directory on the server.

When the humans leave
Well, a plain html url is where it all began. But human edited urls whould require billions of people to constantly sit and re-write webpages everytime you visit the TV Guide or Google or Wired or somewhere else. The contents is simply too alive (dynamic) to have people edit html files and put them on servers. So what to do?
Well instead we get a programmer, preferabilly a webprogrammer to write a program which can generate/create a webpage based on certain details we specify for his program. The details we specify for his program are the ones which makes up for the dynamic contents.
The programmer then writes a program which can read small details and then, based on the details, do some lookup's in database tables and the take that contents from there and put on the webpage with a specific pre-defined style. Ie. CGI which means common gateway interface. A webserver can use CGI to run programs instead of just loading .html pages to the user.


PHP, ASP, PL, JSP ? = and & etc
URL 1), 2) and 4) are all CGI based URL's. And those are the most common way to do things. CGI based urls can use a horde of programs. In the beginning we saw real PC based .EXE files which were run by the webserver. Today it is more common to see .ASP if the webserver is Windows based or PHP if it is Unix based. If you run into JSP then the webserver is hooked up on a Java based engine to execute the programs (Windows or Unix). PL (Perl) is used on both Windows and Unix, but most commonly we see it on the Apache based webserver opposite to ASP which works only on the Windows webserver (IIS Internet Information Server).

CGI based URL's
Have a look url 4). and take notice of the clear red part of the url. Ie. the part which spells membercheck, this is the name of the program written in PHP, ASP, PL, JSP or something else. There are several reasons why the webmasters tend to hide the language used inside the webserver. Because it makes harder for an exploiter to find his way in. Look at the url below and I bet you can answer which language is used and which part is the program

hxxp://nikkisplaymates.com/sd3.php?show=recent_gal_updates

Correct, PHP is the language and sd3 is the name of the program which generates the webpage.

If we look at the url in case 4) we have a part after the domain name. It looks like this:

/membercheck?path=ms4276/streaming&fname=ms4276500k.wmv

We already agreed that membercheck is the program name written in some language. Now this program has to have some details to perform what ever it does to find the proper information for our webpage. Those details are called parameters and in our case we have:

2 parameters:

path
fname

The parameters are seperated by the & sign. Both parameters have values. The value for path is ms4276/streaming and the value for the fname parameter is ms4267500k.wmv.

2 values for the 2 parameters

ms4276/streaming
ms4267500k.wmv

Now, we can only guess that the program (membercheck) uses the path parameter to find the proper directory on the webserver and further more the program finds the video file by the information in the fname parameter. Last there is a fat chance that the program checks that the user is authenticated, as the program name is membercheck.

It is VALUES of the PARAMETERS which you have to figure out, when you make a download list for ReGet Deluxe or OEEE, or whatever program you choose.

Invisible program ?
Well, take a look at url 1) and 2). As we know that the program name is located behind the / for the domain name and before the first ? then we realise that the program name is not shown. It is simply invisible.

This is surely not the case, because there is a program there. Except you do not know the name of it. The reason is that any webserver has some default pages it loads if the user does NOT specify which specific page to load. The most common default pages are:

index.html
index.htm
default.asp
index.jsp
index.php

The web administrator can configure this as he sees fit. So if you find and url as 1) or 2) then the case is just that the webserver is using the configured default file as the program which generates the real webpages.

How to siterip - fooling around with a purpose

Now this is a subject which will be divided into several articles, mainly due to the fact that there is no 100% correct way of doing it and the diversity of sites is huge. So look at this as a learning curve where you pick what ever advice and tips you find useful for your needs.

Define your goal
First of all you need to define your goals for your siter!p (later: SR). What do you want to accomplish ?

• Getting all the pictures from a site ?
• Getting all pictures and videos ?
• Getting everything ?
• Getting just a partial rip ?

If you're not doing an update, as they most often can be done quite easily and often manually then pick your poisen. Personally I aim form completeness.

This means get all media which is of value to you - pictures, videos and other things which does NOT change over time.

Size your opponent
Given you've choosen completeness the you must size your opponent. Is the site a solo model site or is it a larger multimodel site where the contents could spand 10+ dvd's ?. When sizing your opponent there is two things which seems to be of defacto rules.

• Multi model sites = cleaner html/cgi and structure = Autoleechable
• Solo model site = can be dirty = Expect manual tasks

Collect intel
You need to collect intelligence about the site to find out what the best SR way is. Here is a list of things you should make note of while you click around and view the html page sources (because that will be required from you).

1. Is the site using plain HTML and files ?
2. Is the site using CGI/PHP/ASP/JSP/Perl or similar to generate pages ?
3. Is the site using redirection from the pages to the content you want ?
4. Is the site a front cover for a larger contents server on another address ?
5. Does individual picture/video pages link to other pages or is it a "dead end" ?
6. Is the site using Adobe Flash to present contents ?
7. Is the site using JavaScript and pop-up's to present contents ?
8. Is the site providing zip files for contents ?
9. Is contents available in multiple resolutions and qualities/formats ?
10. Is the site based on FORM login or basic HTACCESS ?
11. Does the site require you to re-login after a periode of time ?
12. What seems to be the limitations of simultanious downloads ?
    

1) If you can answer YES to bullet no. 1 then go further and copy/paste random links to
pictures and videos. Put them into NotePad, UltraEdit or EditPad Pro. Copy 2-5 links to pictures from the most recent sets and 2-5 from sets which seems to have been produced at the beginning of the site and 2-5 from the time when the sites was half-old. Go to your editor and look at the url's and detemine: Is there a common structure ? Was the name of the set the same as the directory part of the url ? What tasks is needed to go from set name on the page to files on the site ? The reason is to determine wether or not it is sufficient to just download the media or if you need the HTML also. Downloading the HTML will require you to do more cleaning up after the download, it will also require you to do more structure work when the download has finished.

2) If the answer to this is YES then expect some decent structure. Because the computer generated pages comes partly from a database on the server which holds set names, links and other pieces of information. Do the same as required in 1) but make not of what parts of the url's change from media to media. To get the url's you must be looking at the pictures/videos and rightlickhing choosing Properties and copy/paste the url of the media, possible also copy/paste the url in the address linie of your browser. Copy/paste the url (copy link) which you clicked to arrive at the media. Look at them and determine what steps was needed to go from the frontpage to the media.

3) If redirection is used then primarely make a note (copy/paste) the path of the media and compare them to the link the media came from. Especially make a note of the content is on a deeper level than the page the link came from or a higher page. Deeper and higher is determined by the number of / in the url of the media/page. Higher amount of / means deeper. Also, find out what part of the link url was used to determine the specific redirection - if any special part seems to be the cause.


4) Some sites hosts all their real contents on other servers or on download servers. This has been the case for sites like Only Tease, Brazzers etc. This tends to change as the cost of hosting fluctuates. The download servers seldom require login. Instead they either require a proper cookie or a proper session url. You will have to find out which of the two it is, further you will need to find out how long time the session/cookie lasts before it needs to be refreshed. You will also need to find out if the download server require a proper referer. The easiest way of finding this is to try to use ReGet Deluxe on a link. Look at the HTTP properties in ReGet to look at the cookie/session. And copy the link to a new browser and see if it allows you to download. Remove the refere in ReGet and see if it allows you to download. Most often you just need a proper cookie. Some times you just need the proper name of the encoded url for the contents server. Most often the cookie/session lasts 1-4 hours.


5) If every page with a picture/video has multiple links to the main page, set page and next picture/video etc. then find the urls which links further and copy/paste them so that you can filter them away during your download.


6) If Flash is used to present the contents then you're in for a 100% manual download - more or less. None of the current tools handle Flash sites very good. So get your patience and long hours ready.


7) If JavaScript and pop-up's are used the examine the code and the urls from link to media to see if you can avoid having to execute the JavaScript to download the media. So far I have only seen 1 site that was un-circumvental. If you cannot avoid the JavaScript then you're in for a manual download.


8) If the site provides zip files for the contents then make sure that there ALWAYS is a zip file. Also make sure that it is the proper zip file. Most often the zip files are pre-produced (except for 1 site - to my knowledge) so you will have to probe a few zip files to see if the zip everything or if they are sloppy. Brazzers sometimes zips thumbnails alongside the contents as example. If the zip's match the individuals then go for the zip's otherwise omit the zips and go for the individual downloads.


9) Always aim for the higest quality. So if the pictures come in 2-3 resolution then choose the higest resolution. If the videos comes in same quality and bitrate but multiple formats then pick the one with the lowest MB. That is most often WMV. MPEG/MOV tends to be larger. Download both the AVI and WMV version and watch the video to determine the better of the two.

10) FORM login is the login type you see at Hotmail, YouTube etc. These days the FORM login often comes with some OCR to accomplish the username and password. If FORM is used then be certain that a cookie/sessions is the key to accessing the site from your download tools. If HTACCESS is used then you should be able to go directly to the members page with an url like http://username:password@www.sitename.com/members/ or what ever the URL is. Now make a note that Internet Explorer does not allow that method for logging in (unless you tweaked it), so try it with FireFox or your download program.


11) Re-logins is used with FORM sites and it refreshes/renews the cookie/session. So if you queue up a huge download which will take several hours then find the time limits and do not queue up more than you can manage to download with the time limitations. Why?, because otherwise the site admin will see you hammering the site all night and the chances for a ban is severly increased which means you will be cut off.


12) Some sites has limitations to the number of simultanious downloads. For Ann Angel as example there is a limit of 2. This means that you should not exceed this number or you will be cut off by the site. Choose a number which fits a hardcore download guy using FireFox or some other tool. That means go no higher than 8 simultanious downloads. Go down to 2-3 when you start your download manager and watch it progress to see if it could be increased. Do NOT roll over a site with 32 simultanious like a steam roller. The site admin or system will notice and you will get a ban. Nobody likes their site to be 'raped'.

Probing
Now it is time to test wether your intel is sufficient. Unless it is a video site then try with the intel you have on the pictures first. This means that you will need to make a new project in Offline Explorer Enterprise Edition and put your urls, cookie=, referer=, username and passwords into OEEE and see wether it will download outside your Internet browser. Start with 1 set at the beginnig. Set your URL filters as tight as possible according to your intel, both for server/domain, path and filename (to avoid downloading thumbnails). Limit your file types to pictures only. Hit Download and switch to the queue tab to see if things are resolved properly and that you infact do download pictures. If NOT, then relax your restrictions until you get an acceptable result. Next try with 2 sets and see if it works. If it does then queue up the lot and let it run. Still taking into condiration any time limitits and simultanious limitations. Use F9 to pause you download when needed to watch the url resolving. And use # as a comment to play around with your urls.

cookie=user=myself&pass=secrets&sessionid=abch48d1gaup39Hl
referer=http://www.site.com/members/image?setname1
http://www.site.com/members/image?setname1

#http://www.site.com/members/image?setname2
#http://www.site.com/members/image?setname3
#http://www.site.com/members/image?setname4

It is a good idea to configure OEEE to either Download All Files or Do not download existing files because while you are probing you may get some "trash" along the download. Try to limit the amount of excessive stuff you download. Clean you download folder after probing has ended and before you begin the real deal.

Finale
To get good download lists then you may need to save some html pages and possibly use some regex or some column copy/paste. Macro downloading may also be of benefit to you. See articles ablut MACRO download and regex.

ReGet Deluxe can import a text file with URL's so make one for the videos and import it. Mark all the url's in ReGet and set the properties for the download path, cookies, possible username/passwords etc, before you begin. Set max simultanious downloads in ReGet to 8 (on huge videos).




The article serie will continue with more specific information and tool practice alongside regex and download list creation. Until then - take care.