Wednesday, January 13, 2010

PServeCheck v4.5.2.50 new features



The most popular collection management tool PServeCheck, responsible for 47% of all downloads from The Vixen Grimoire and used by 25% of all TVG users has gotten new features. Features you can benefit from.

Some of the hot features are:

  • New leech methods supported (Pictures only / Videos only)
  • Estimated remaining time for various operations (Check, Hunt and Move/Copy)
  • Speed calculation of various operations (Check, Hunt and Move/Copy)
  • Possibility to show/hide burned collections
  • Smart caching routine. Calculates CRC values of files when they are downloaded by PhotoServe (Set in Cache tab in 'Settings->Preferences')
  • Offine collection can now be filter
  • HD Backup supported
  • IrfanView support added (Currently ACDSee Classic 2.43 or IrfanView)
  • Leech channel view. See where collections are set to be leeched. Leech settings can be changed by right click 'Other properties->Leech in'
  • User defined frequency of CSV check on repository (default 20 mins)
  • Auto enable leeching of collections if their value is lager than x (x defaults to 250 bu can be set by user)
  • Media location is now shown in "Backed up" column
  • Media location can be user set when setting collections to 'backed up'

PServeCheck is still only 3.5Mb large and handles collections like a charm.



User Story: One of the things I enjoy most with PSC is that I have set an entire collection as 'Auto Setup', 'Auto Update' and 'Pictures Only'. This means that I do not have to worry about anything. Every time there is a new csv in the group i like, on TVG, then it is automatically downloaded, and setup in my collection manager. PSC automatically begins leeching the pictures and it stops when all pictures are there and checks the collection afterwards. Done deal - 100% automatic, I like it :D


User Story: To help my community I have set 'Auto update if stat value > 200'. That way PSC automatically takes care of supporting the community for the rare stuff.



User Story: With the new 'Leech channel view' I had a good overview of where things were configured to be handled. I was also able to change the channels easily by using the 'Other properties' settings for the collections.

Wednesday, December 23, 2009

Vixen Grimoire and Guild



Here a holiday specal heads up for those of you who wonder why and what the names mean.



The Vixen Grimoire



Maintainers Guild

Here is the technical explanation:

Vixen
Vixen is used as an epithet to describe a woman:
Vixen, a woman noted for her sexual attractiveness


Grimoire
A grimoire is a textbook of magic. Books of this genre, typically giving instructions for invoking angels or demons, performing divination and gaining magical powers, have circulated throughout Europe since the Middle Ages.

Magicians were frequently persecuted by the Church, so their journals were kept hidden to prevent them from being burned.[1] Such books contain astrological correspondences, lists of angels and demons, directions on casting charms and spells, mixing medicines, summoning unearthly entities, and making talismans. Magical books in almost any context, especially books of magical spells, are also called grimoires.


Guild
A guild is an association of craftsmen in a particular trade.

The earliest guilds were formed as confraternities of workers. They were organized in a manner something between a trade union, a cartel and a secret society. They often depended on grants of letters patent by an authority or monarch to enforce the flow of trade to their self-employed members, and to retain ownership of tools and the supply of materials. A lasting legacy of traditional guilds are the guildhalls constructed and used as meeting places. The modern patent system was set up to break the power of the guilds.

Tuesday, December 22, 2009

PSPro Verify v1.01.89 changes


PSPro Verify has undergone a bunch of minor fixes and some additions. Read the changelog below and findout what is new.


PSPro Verify is still among the best tools there exists for collection management and is extremely versatile and customizable.


As you may know PSProVerify originates from the programming language Perl, which makes a foundation for porting the application to other platforms than Windows.


Among the very slick features is the new Advanced Collection Manager which is able to automatically organize your collection on your harddrives by rules.


Also the support for HDD backup is among one of the great new additions.




  V1.01.xx
 
  --------
 
  PSPVerify Released.
 
  Added (00):
  support for the new CSV Repository System (aka Mirror2)
  Changed (01):
  better automated csv upload handling.
  Added (01):
  faster writing of trigger config.
  Added (02):
  Updates from CSV Repositories are always handled. Even if the local csv is considered better. That means a Final CSV can be replaced by a Ongoing
 
  CSV from the remote Repository. This can be deactivated (in Settings) but it is recommended to leave this setting alone.
 
  it does only affect the latest changes (the ones that are received after installation of V1.01.02) only the csvs
 
  that are uploaded from today on are updated. it is not about synchronizing with the very old stuff that was updated years ago
 
  on the repositories.
 
  this feature is about beeing able to get a (final) back to (ongoing) - only when someone uploads a csv
 
  the only problem could be that you reset the "last-updated" counter for the csv repositories - or add a new one (with a new name)
 
  then you would probably get the last 2000 entries compared
 
  an turned off 'C' flag (auto update CSVs) in PSPVerify should not update a csv - no matter what the other option is set to.
  Changed (03):
  Min time for checking for updates reduced to '0.1' (that is 6 minutes) - as the update does not waste any bandwith if nothing is new.
  Added (04):
  Blocking Bittorrent Leeches for triggers, selectable from the menu is 0.5 to 7 days. For that time no Bittorrent Leech is started automatically.
 
  The block can be removed with 'UnBlock' at any time. To be used for bad leeches that are constantly restarted because the tracker reports a
 
  wrong count of seeds or if the remote csv is found to be bad. Automated compare compares only by size and name of files not crc.
  Added (05):
  Old/Unused entries from the triggers are removed during a triggerupdate.
  Changed (07):
  Major Change of the way PSPVerify writes and reads INI/TRG files, now it runs in sync with mirc read and writes, this can possibly lead to
 
  blocking calls (freezing up) while mirc locks files (and freezes).. However this new way of handling the writing should ensure that no data is
 
  lost during asynchron write accesses.
  Fixed (09):
  A Nasty bug causing 'HASH' entries in the Ghostfile is fixed now. All bad entries are removed automatically.
  Added (11):
  'Hide Burned' Button added
  Added (11):
  ACM 2.0 - New ACM Organizer - See the HOWTO at the end of this README.
  Fixed (11):
  Movecollections cleans remaining empty folders up.
  Added (12):
  Torrents to be removed are identified by names in uTorrent (over the WebUI) too! This will remove all torrents with the same name, even if the old
 
  .torrent file cannot be accessed (any more) - which has lead in the past to double torrents in the client.
  Fixed (13):
  The Options/Settings windows are scrollable.
  Added (14):
  after every Check (not the Readonly) that has 0 missing files, all files are deleted in "removed" folders : _Wrong_crc and _Wrong_size .
  Added (15):
  removed folders : _Wrong_crc and _Wrong_size folders are also deleted when a collections completes (even with read only check)
  Added (17):
  partial PS leeched file status are displayed in the Report.
  Added (19):
  priority and leech block calculation has been adapted to the new Part Leech System of PS
  Added (20):
  Bad_CRC files are moved to the PSTEMP dir and not to the Removed Folder
  Added (20):
  nerocmdoverride setting to specify the nerocmd.exe directly instead of looking for it in the Nero folder.
 
  example: add the line to PhotoServe\Settings\PSPVerify_Config.ini
 
  nerocmdoverride=C:\Program Files\Common Files\Nero\Lib\NeroCmd.exe
 
  do this while PSPVerify is not running.
  Added (21):
  the debug window shows information about the nero process when trying to burn
  Added (22):
  the nerocmd is found for Nero8 automatically. However Nero8 seems to have a bug and burn does not YET work. (see the switch 2 'Addeds' later)
 
  FIX: http://www.nero.com/enu/downloads-sdk.html (its free)
 
  download the sdk and replace the nerocmd with the nerocmd inside the SDK
  Added (22):
  protectburnedcollectionwith param added. This is hidden because it will freeze PSPVerify for the full rar process totally.
 
  that's why you should not use it, then you cannot complain. This option will RAR up the full collection folder and put 1GB rar
 
  files to the DVD instead of the files themselves.
 
  example: protectburnedcollectionwith=C:\Programme\WinRar\WinRar.exe
  Added (22):
  switch to disable the usage of the param file for burning. instead all params from the param file are put in the command line.
 
  use this switch to make Nero8 work, but use it at your own risk since the command line size is limited and params can be
 
  truncated. This risk is high if you burn many small collections with many paths and csv names to be put in the command line,
 
  burning a DVD sized Collection will work in almost all cases.
  Added (22):
  a new remote command to write Nero Images. Intended use is to write every night a few of the Disc Images to be burnt.
 
  Usage: I have created a .bat file with 1 command inside:
 
  echo >P:\Programme\mirc\PhotoServe\Settings\Checker_RC.txt burn DVD\d+ r 2 Z:\Burn_IMAGES
 
  This will make PSPVerify create 2 Disc images of Ready-TO-Burn collections that are DVDxx triggers (no CDxx)
 
  If I put this batch file into the task planner to be run at 3am I will find every morning 2 NRG files in Z:\Burn_IMAGES
 
  and can burn them. This works especialy good with the 'protectburnedcollectionwith' parameter in PSPVerify.
 
  Note that the Target Path may NOT CONTAIN SPACES.
  Added (23):
  The compression of Collections is calculated and not 'guessed'. Thus eliminates the need to compress the Collection
 
  and simply store it with maximal recovery records.
  Changed(24):
  Minor change to display the Versions of NERO on start.
  Fixed (25):
  Bad option `-text' at ....
  Fixed (26):
  Minor problems all around ..
  Added (27):
  2 new config settings - a Trigger can be turned to auto-update (temp or permanent) if the stats value of a collection
 
  is above a set limit (Settings -> Misc Options -> Default Trigger States)
  Added (28):
  AutoCheck will stop checking (after it finishes the current Collection) if user moves the mouse (IDLE < 100)!
  Added (29):
  Hunting into a leeched Collections causes an interruption of the leech -> requeue -> releech (required PS4.9b34 or above)
  Changed (29):
  Hunt program code rewritten
  Added (30):
  The reports show the absolute numbers of parts as well for partial downloads
  Changed (31):
  reports lists files "unsigned" decimal
  Fixed (35):
  When moving collections and there was not enough space on the target drive the collection would be left turned OFF
  Added (36):
  When creating csvs:
 
  * problems like double spaces will be correct in the filesystem.
 
  * there is a switch to correct smaller uglinesses in file paths (like a space in front of a coma)
 
  * there is a swtich to extract all files in a found zip file before creating the csv (so that the csv contains the
 
  zip content instead of the zip file. The zip file is extracted and deleted, so dont use this setting if you build csvs with zip files)
  Added (37):
  Primary Support for burn-tool ImgBurn from www.imgburn.com
 
  ImgBurn is a freeware tool that supports a lot of command line switches and is now used in PSPV.
 
  Nero should still be working (i hope) . But it is untested .
  Added (37):
  2 new limits in the BurnDialog:
 
  * you can now limit the amout a disc is filled to prevent it from being offered in the dialog
 
  (a level of 85% while set to 4480MB will not create any DVD that is not at least filled 85%)
 
  * you can also limit the total amount of discs that are offered in the dialog
 
  (if set to 8 and even if you selected 100DVDs to burn, will only create the first 8 (best filled discs))
  Changed(37):
  The way the automated burn (image create) works now has been changed.
 
  echo >P:\Programme\mirc\PhotoServe\Settings\Checker_RC.txt burn CD\d+ r 3/9 D:\Burn_IMAGES
 
  creates 3 discs (of the current setting in the burn dialog) up to a maximum of 9 discs in that folder
 
  you can mix any CD or DVD or even the Pic-Only and Vid-Only Discs
 
  the 2 new limits apply also to the auto creating
  Fixed(38):
  Works with Nero again
  Added(45):
  http ignore when a server is down
  Added(46):
  new Collection Tool "Attach Backup Drive Directory"
 
  since you can have a backup Harddisk full of collections you would take hours or days to restore them.
 
  to make it quick you can use this feature to restore your collections in one command.
 
  this does the following:
 
  *) search 1 directory you specify for collection folders (all in flat dir structure, this means no group subfolder or anything)
 
  example: you specify F:\Backup Dir\ -> this will find F:\Backup Dir\SNCD1, F:\Backup Dir\SNCD2, F:\Backup Dir\SNCD3, ...
 
  *) if the directory name matches EITHER a trigger OR a csvname OR a collection name it will attach the folder to that collection
 
  ONLY IF the collection has no valid path. this means if the collection is turned off or not, the path of the collection MAY NOT EXIST!!
 
  *) turn on the collection (if it was turned off)
 
  *) download the latest CSV ! it does this - if possible - ALWAYS! if it cant find a csv it will be a problem you have to solve
 
  *) check all freshly attached collections to make sure the count is correct. Since there is no cache this can take a long time.
  Added(47):
  added "eject after burn" switch for ImageBurn
  Added(48):
  a new option in "Config -> Settings -> Default Trigger States" : "turn on autoupdate when last trigger (of a series) completed"
 
  this feature checks for SETUP and INCOMPLETE collections, and if there are 0 collection with autoupdate on it will turn on the first found
 
  trigger. Please not that this does not mean it will turn on the next trigger after the completed (and this is a smart feature).
 
  Example. Triggers series "abcd" -> you have setup
 
  abcd1 with 123/123 files autoupdate 0
 
  abcd2 with 120/120 files autoupdate 1
 
  abcd3 with 0/12 files autoupdate 0
 
  abcd4 with 0/13 files autoupdate 0
 
  abcd5 with 130/130 files autoupdate 1
 
  abcd6 with 150/150 files autoupdate 1
 
  abcd7 with 123/124 files autoupdate 0
 
  this will turn on abcd3 ! because it is the first trigger that is incomplete !
 
  if abcd3 OR abcd4 OR abcd7 are set to autoupdate nothing would happen.
 
  since abcd3 is turned on now, nothing happens until the last of all autoupdate triggers (in this case abcd3) is completed.
  Added(49):
  automated way to upload samples to csv sites that support this.
 
  You have to configure your csv_reps.ini file (see your prefered website for details) to have c10 and c9 fields.
 
  Then select several triggers (that are online and have files) and press CTRL+SHIFT+F8 to start the automated upload process.
  Added(51):
  New switch for "turn on autoupdate when last autoupdate trig of series completed" to limit this function to FINAL and REBURN csv
  Added(52):
  If both settings "turn on autoupdate when a collection gets a csv that is final/pre-final" and
 
  turn on autoupdate when last autoupdate trig of series completed are turned on, PSPV will not setup a new csv as
 
  AUTOUPDATE unless there is another trigger in that collection series already set to AutoUpdate
  Changed(53):
  OFFLINE collections are skipped in AutoUpdate count for collection series.
  Added (54):
  As a test - you can disable the cache for Creating CSV - lets see if this helps with crc problems that someone is experiencing ;)
 
  (settings -> Misc Options -> CSV Create Options -> no cache)
  Added (55):
  Statsbot values are now interpreted as per byte values. The value will change when a new csv comes in. The value shown is for the csvsize
 
  so it will not change when you leech. Also the sort by value will sort on the per byte basis, because otherwise a low valued DVD will always be in
 
  front of a low valued CD even if the DVD is much more worthless.
  Fixed (56):
  A very nasty bug, where the crc calculation for creating csv was completly destroyed. (Introduced in .54)
  Fixed (56):
  bug disabling the Sort by Value Function .. sorry this took so long to find. Do "Stats Refresh" to get rid of the insane values.
  Added (56):
  new switch to disable the per byte basis for the statsvalues. All collection are assumed a fixed size (that means the values are much higher for CD collections, but
 
  on a value per bytes (on the disc) basis it is more correct)
  Added (58):
  AutoHD Backup can be limited to collections with final csvs.
  Added (59):
  Extra Info is now added when getting a csv from a channel.
  Added (60):
  A little ASCII Graph in the report, if Transfer Stats are Available.
  Fixed (61):
  Bug that showed random stats for triggers in reports
  Fixed (62):
  Small bug that caused the hostbase to be not properly detected for http requests
  Added (63):
  Another cool feature when creating self-maintained csvs. Selecting "Align Files" from the "self-maintained" Menu will do this:
 
  *) check the size of the selected Trigger files (in the collection folder)
 
  *) move oversized files (sorted by alphanum, so use date-styled dirs) to the next trigger
 
  *) create a new dir for the next slot (the slot has exists)
 
  *) set the full slot to Pre-Final
 
  *) start a csv-create process for all "touched" slots
  Fixed (64):
  Bug with Align Files where the sorting of the dirs did not work
  Added (65):
  Menu item for 'Self-Maintained' to quickly change the state of a CSV (Rename CSV) without any further activity.
  Fixed (66):
  Reports dont show empty transfer statistics.
  Added (67):
  Added a few more Windows-Ways-To-Find-Applications-For-Typeextentions ! Good work to have an Operating System have ~30 different methods !!
  Fixed (67):
  ACM Bug when selecting target drives space rules
  Fixed (67):
  Download of Stats from 'strange' urls
  Changed(69):
  Improved internal search algorithm in Trigger Files => Better performance for all Operations
  Fixed (69):
  Fixed a minor bug in the 'Align Files' function
  Added (70):
  Added an ignore file pattern field for files inside archives. Files in Archives matching will not count and
 
  archives with only these pattern will get deleted. <10k are deleted directly others are deleted to Recycle bin (if enabled in PSPV)
 
  a good pattern is '(\.txt|thumbs?\.db)$' which will ignore all '*.txt, thumb.db and thumbs.db' files ;-)
  Added (71):
  Added 'Correct Changed Drive Letters' in Menu 'Collection Tools' :
 
  the commands checks ALL online collections for existing report, csvfile and the collection path.
 
  If one item is missing it looks on all existing drives if the path/file exists there and
 
  changes the setting if a positive test is done.
 
  This is very useful when a drive letter is changed, since all paths are just the same but only
 
  the drive letter is different.
  Fixed (72):
  A bug where the Align Files function does not work if there is only 1 further trigger, which was ignored.
  Added (73):
  FTP Upload works now where available
  Added (74):
  Support for Torrents to be redownloaded from the tracker after upload, if the Tracker added some extra config.
  Added (75):
  The files for created torrents are now sorted by date! This will help to create a new torrent with all files in the same
 
  order as any previously created torrent would have been sorted. Which will give a new torrent that can be 100% from the
 
  previous torrent. Unless the dates are wrong...?
 
  (This will however not help when a older file is replaced,
 
  and because you can never know which file might be replaced in the future this can never be done.)
  Added (76):
  PSPV supports Autoconfig from BT Trackers (there is a button in Settings -> Bittorrent -> Autoconfig
  Changed(79):
  FTP Upload has now a timeout of 60seconds
  Added (79):
  When checking a BT Leech Locked collection to find out if its complete, PSPVerify will resize all files that are too large and check the crc again.
 
  This is a workaround of a bug in uTorrent where uTorrent creates a few extra bytes sometimes.
  Changed(80):
  Many tweaks of values
  Added (80):
  New option for burning UDF in Burn Dialog for ImgBurn.
  Added (81):
  CSV and CSV path are split to two columns.
  Added (81):
  Added two new export menu entries. You can now export the viewport content (that is everything displayed)
 
  to the clipboard. You can export all displayed lines or just the selected lines.
 
  The entries in a line are separated by TABs and the Lines are separated by newlines.
 
  This content can be pasted to any known Table Calculation Tool.
  Added (81):
  A second parameter can be supplied when starting PSPV (where the first is Mirc-Dir, which is the base dir)
 
  the second param will override the Triggers Location Path. This allows to run a second instance of PSPVerify
 
  in a different location, while it still uses the same Trigger config as the other instance. Since PSPVerify
 
  uses a file-lock technique it is safe to change settings in multiple instances and the files will not be harmed.
 
  This is usefull to run some jobs like burning Discs in 1 extra PSPVerify instance while the other one
 
  keeps running unblocked.
 
  NOTE: some settings are stored in the ghostfile and those are unique to each running instance.
 
  NOTE: you can't use this to run several leeches in multiple Networks in PhotoServe
 
  EXAMPLE: run 2 PSPVs (1 will use all defaults in C:\Mirc, while the other 1 will use its own path and
 
  the same triggers)
 
  > PSPVerify.exe C:\Mirc
 
  > PSPVerify.exe C:\Another-Mirc C:\Mirc\Photoserve\Triggers
  Added (82):
  when using utorrent there is no longer a read-only check to find out if a BT leech is complete. Instead the
 
  WebUI interface is queried, and only if seeding is returned for the torrent, the read-only check is performed.
  Fixed (83):
  0 bytes files are no longer included in the torrent.
  Fixed (84):
  no archives are deleted when they are larger than 10k (as password protected archives are falsly detected as empty)
  Added (84):
  Added an option to clean up the "removed" Files folder, you can set how often to check and how old the files have to be
  Added (85):
  you can create your own entries in the Collection Tools -> Tools Urls Menu .
 
  adding a line to the [config] section of the PSPVerify_Config.ini will add an entry to the menu, the format is
 
  toolurl.=,,
 
  The appears nowhere but has to be unique. is shown in the menu, and is
 
  the url to be opened in your browser.
 
  Example (put the line below into your PSPVerify_Config.ini, somewhere into the [config] section.)
 
  toolurl.pspv=http://pspv.dyndns.org/,The PSPVerify Homepage
 
  !! NOTE : you may only edit the PSPVerify_Config.ini while PSPVerify is NOT RUNNING !!
 
  NOTE: Currently supported modes are (you can "OR" them):
 
  0 = leave the url alone like it is
 
  1 = add incomplete triggers to the url (in this format: trig=trigger1,trigger2,trigger3
 
  2 = add incomplete series to the url (in this format: series=series1,series2,series3)
 
  4 = add setup series to the url (in this format: series=series1,series2,series3)
  Added (86):
  the crc32 calculation is now "thread safe" - maybe this fixes all crc problems people had in the past
  Added (87):
  the AutoCheck (if enabled) does additionaly a quick "count of files" every 10 days for all
 
  online and complete collections. a count of files takes a few seconds and it will basically just
 
  check if the dir is missing any files.
 
  this check is only done when you are idle and it will stop when you move your mouse.. dont worry! DO NOT FREAK!
  Added (87):
  When PSPV is interrupted during moving a collection it will resume the move the next time it starts.
  Added (87):
  Copy Collection has a switch to copy the csv. The target is the same location as the csv is, just on the Backup Drive.
  Added (88):
  PSPV AutoCheck checks online collections with a FAIL (or other) status. Because only offline collection should have a status of OFF
 
  everything else is some kind of "problem" and should be checked.
  Added (88):
  new switch in Settings : "Protect with WinRar" (needs a restart).
 
  If it works PSPVerify will display a log entry during startup: "All collections will be compressed before burn with ..."
 
  Burned collections can be protected with WinRar (which has to be installed). Instead of burning the
 
  collection folder PSPVerify will create a rar with as many recovery records that fit on the CD/DVD. A recovery record can
 
  be used when a block on the CD/DVD becomes inaccessable. That way you can fully restore a collection from a CD/DVD
 
  even if you cannot read some of the data on that disc. (To recover a broken rar file you have to select "REPAIR RAR"
 
  in WINRAR Application. It will "repair" the rar to a new location and you can extract all files from the restored rar
 
  or you can reburn the restored rar to another DISC)
  Added (88):
  Some new reports details
  Added (88):
  Added "Hide Incomplete" Quickfilter Button
  Updated(89):
  Import Settings

Wednesday, December 16, 2009

CSV usage - collection manager tasks (newbie)


If you have read the articles about CSV files - an introduction and CSV - stages of evolution, then you now know that csv files are used to index your files into reasonable chronological chunks, most often DVD chunks.

This article is about which tasks there is in organizing your files so that they match those csvs. To utilize csv files you must use a program. Much in the same way you would a BitTorrent Client and a .torrent file. The programs in question are Collection Manager tools. This article is not about a specific collection manager program, but instead of the general tasks within these programs.´

Directories:
There are a minimum of 4 directories involved with a collection manager. The first one is the directory where you will keep all your csv files. This directory is referred to as your csv folder. The second directory involved is the directory for your reports. Reports are text files which hold the result between your multimedia files and your csv files. The csv folder and the report folder need to be 2 different places. The 3rd directory involved is your collection folder. The collection folder is the place where you keep all your CD's or DVD's, this is the folder where your Hegre.-Art-DVD41 folder and data will reside. The last folder is your download folder. This is where you dump all your data before they get organized. Stuff from usenet, websites and every other place where you get data from.

  1. CSV folder
  2. Report folder
  3. Collection folder
  4. Download folder
You will need to configure your collection manager for these 4 directories as a minimum. You can have multiple collection folders aswell as multiple download folders. Start off by copying your csv files to the csv folder. Empty the folder for the reports, copy what you do have into the collection folder in the proper sub folders. Do a little cleanup in your download folder.

Setting up the first collection:
The next thing is to find a csv for some of the stuff you do have and know should match some files. Let's pretend that you know you have some files which supposedly should belong to Hegre-Art-DVD41. So find the Hegre-Art-DVD41 csv on a csv site and download it. Use the collection manager to setup a new collection with the csv you have downloaded. And point the collection towards the directory where you have stored the files from Hegre-Art-DVD41.

Check:
Depending on the tools of choice then the collection manager will check the files you have in the directory, against the files specified in the csv. The csv for HGRARDVD41 (the trigger name for the csv in question) has 1673 files registered. After the collection manager has checked the files on your harddrive, then it will have show you how many files you have complete, compared to the csv. If everything is there and match 100% then you get a count of 1673 and a missing/wrong count of 0.

If you get a missing/wrong count which is above 0, then some files are missing or not the original files which matches the csv. This means that even though you may be able to view the picture or watch the video, then someone has altered something in the file. Theas files will me moved to a directory outside the collections directory. If you are lucky then the files which does not match the csv you are checking, will match to some other csv, let's say HGRARDVD40 as example. The directory where the bad files reside in is a good source as a 'hunt' directory.

Hunting:
Hunting is the term used to scan and look for files which match a csv. Often you would hunt from your download folder, which possibly contains alot of ramdom files, jpg, avi, mov, rar, zip etc. The files which are discarded by a check may match other csvs. You will have to empty you "bad files" directory from time to time, unless you have unlimited space and wants to keep stuff which does not match anywhere. Modern Collection Management tools can look inside zip and rar files. So there is no need to extract the files yourself, unless there is some damage to the archive. You can configure the Collection Management to hunt from any number of files residing on multiple sources. When hunting the Collection Management tools are able to delete a found file from the hunt directory if it matches a csv, thus cleaning your hunt folder while hunting. This ability also goes for the files inside zip and rar archives.

Reports:
Everytime you check or hunt then the Collection Management tool will produce a report. This text file will show detailed information about which files are good and which files are bad. Also information about sizes can be found in the report. The report can be used in various ways, both by the tool and you. The reports are especially useful when you have collections on offline storage.

Saturday, November 14, 2009

FlashGot - a beginners guide to easy downloading with Firefox

FlashGot is the best Firefox download manager integration around. That is their words and they may very well be true. FlashGot ( http://flashgot.net/ ) is a tool to integrate the website with a download manager of your choice.

In regard to download managers then we are in business with tools like DownThemAll ( http://www.downthemall.net/ ), which is free and the commercial tool ReGet ( http://www.reget.com/en/ ). I will demonstrate both here in this micro tutorial. We will also have a quick look into some of the configuration possibilities of FlashGot.

Quote: "Spend less time downloading and more time on organizing". 

The image screenshot below shows the Preferences for FlashGot. From the red marked circles you can see that I have a possibility to choose the default download manager as either DTA (Down Them All), ReGet or the build in browser download capability. I am most familiar with ReGet as it gives me the most information and control, but the choice matters little as both tools manage to download properly.



FlashGot supports a wide range of download managers whereas the most popular probably is DownThemAll, FlashGet, GetRight, jDownloader and ReGet.

FlashGot basically has two modes, either you choose to download 1 link or you choose to download the entire page, and everything that is linked to from the page, one level deep. Let's see how things looks using DownThemAll and choosing 'FlashGot All'.


If we switch the default download manager to ReGet then we get a slightly different picture although they are quite similar. With DTA some files are checked as which to download. With ReGet all files are picked as to be downloaded. Both tools require you to hand pick which, because in this case you may want only the default size video and the HD largest quality video.



With both DTA and ReGet the next step is to click OK and let the downloading begin. If you right click a link and choose 'FlashGot Link' which is the single file download mode then you get thrown directly into the single file download mode of the download manager you have set the preference too. Again it is the OK button and your download starts.

The force of download managers like DTA and ReGet are their ability to stop and resume huge downloads with multiple threads. Downloading a 2.18Gb file may take a little while and the best way is to use a tool like DTA or ReGet. Give it a try.

CSV (Final) - stages of evolution


This is a followup to the article "CSV files - an introduction"

CSV files or ecsv files to be specific follow specific steps of evolution. These steps are to complete the csv file so that it ultimately holds a number of files which consume the maximum amount of space on a storing media as a CD or DVD etc. Furthermore the stages have the goal of keeping some kind of chronological order.

Ongoing
The stages are set by the csv maintainer, more about that role in another article. The maintainer starts off by having an ongoing csv. Sometimes this is explicitly  shown when a part of the file name being (ongoing). Most often there is no explicit naming convention to an ongoing csv. This is the case with ongoing csvs from TVG (The Vixen Grimoire) and the former ET (El-Toro).

Pre-Final
When the maintainer has collected enough files to fill a CD or DVD then the maintainer sets the csv to the stage (Pre-Final). This is to signal people using the ongoing csv that it is almost complete and they should expect that a new ongoing csv will soon emerge. According to community rules the maintainer must keep the csv as Pre-Final for a short while to allow other to check the csv and report errors. If there is an error in the csv, ie. wrong set or wrong naming or missing sets then then maintainer will fix the problem (hopefully) and create a new (Pre-Final-1) csv. This can go on for a long time, but it is seldom seen that Pre-Final's go beyond (Pre-Final-4). Once a csv reaches the Pre-Final stage it will NEVER go back as ongoing. This is breaking the rules, and big no-no.

Final
If the maintainer has not had any reports of errors in the csv and feels that the csv is ready to be burned to a CD or DVD then the maitainer will start the next stage, which is (Final). The (Final) means that it is now safe to burn the contents of the csv to a DVD and the user should not expect any further changes to the csv. If the world was perfect then nothing more happends. Everyone using the csv, burn it to CD or DVD and is able to clear up some space on those HDD's.

Re-Burn
But this is not always the case. Sometimes the maintainer is too fast to go from (Pre-Final) to (Final), sometimes people notice errors too late. This is where things get fuzzy, so sharpen your senses. If the change to the (Final) csv is structural only, ie. new layout or new naming or some corrupt picture was removed for good then the maintainer can issue a (Re-Burn). A (Re-Burn) means that the contents of the previous csv is just to be burned to a new CD or DVD. People having the previous version has not lost anything. (Re-Burn)'s can also go into stages like (Re-Burn-1) and so forth.

Final-X
If the change after the (Final) stage is of non structural nature, then maintainer has to issue a (Final-1) and provided there are more changes later it goes to (Final-2) etc. It is seldom seen that anything go beyond (Final-4).

In rare cases the maintainer will choose to use (Re-Burn) as a final exit for the csv if it has reached (Final-4), because it is flat out embarrassing that the maintainer cannot manage make a proper. This last exit is seen, but it is not the way to do it.

There is no stage beyond Re-Burn-10. Pre-Finals never become ongoing. Final's never go back to Pre-Final stages and Re-Burn's never go back to Final's.

It is a common misunderstanding that the use of Final-1, Final-2 etc is whenever the website in question revamps or redesigns itself. This is NOT the purpose of the Final stages. If a website redesigns and introduces all new resolutions of old contents HD images and HD movies then the new revamped site will most likely get a csv set entirely. See with CrushPhoto and CrushPhoto-New as example.

Friday, November 13, 2009

CSV files - an introduction


Recently I was brought to my attention that a simple entry introduction to CSV files in our context was lacking. This article will try to provide you with an introduction to what it is all about.

Facts
  • A csv file is a ASCII text file you can read in NotePad or similar
  • The extension for csv files is .csv
  • CSV means: comma seperated values
  • The format used on Vixen Grimoire, El-Toro, WSC etc is not csv, but actually ecsv
  • ECSV means extended csv
  • The purpose of an ecsv file is to be able to organize your files
  • You organize you files according to an ecsv file with a program
  • Programs for organizing is called Collection Managers (click the link for an intro to those)
  • ECSV files have 1 strict format
  • ECSV files has the file extension .csv
  • ECSV files are commonly referred to as just csv files
  • ECSV files are maintained updated by 1 person at the time
  • It is not allowed to tamper or change the ecsv file made by someone else unless there has been given permission to do so. If you get permission, then YOU become the maintainer.
 A csv file (common name) has 1 file listed on every line in the file. Each file has some values to it. The values are in the following order (inside the csv file)

FILENAME, FILESIZE, CRC32 HEX VALUE, PATH, OPTIONAL COMMENT

An example could be:

img_0750.jpg,438648,A2DF81D5,\2008-11\2008-11-27 - Kelly M\,

The line above tell the following. There is a file, which should have the file name img_0750.jpg. This file has the file size of 438648 bytes and the calculated CRC32 checksum value must be A2DF81D5 (hexadecimal presentation) for the file to be the right file. The file should be located in a folder structure which must be \2008-11\2008-11-28 - Kelly M\. There is no comment to this file.

This means that there is only 1 file which actually matches to be the correct file for that line in that ecsv file. In general the uniqueness is assured by the use of the CRC32 checksum calculation  which is almost perfect for distinguising files from each other.

If you have a picture and you calculated the CRC32 value for the picture it may turn up to be (as example) 52AC2E11. If you then change just 1 pixel and save the file again then the file will NOT have the CRC32 value of 52AC2E11. So the two files do NOT match. Same goes if you change the resolution or scale the file.

The files listed inside the ecsv files on El-Toro, Vixen Grimoire and such places have the CRC32 value of the original files from the websites. NOTHING has been changed, not even a meta EXIF piece of information.

Every csv file itself has a CRC32 value, so it is not possible to change an existing csv file and then claim it is original if the CRC32 values of your file does not match the CRC32 value of the file on the csv sites. Here is an example:



The csv file has the name ErroticaArchives-DVD35(Pre-Final)_1872.csv. 1872 lines with file descriptions is listed inside the csv file, of which 88% is pictures. If you add up the file sizes of all 1872 lines the it amounts to 4,439,771,784 bytes, which is ~94% of a single sided DVD. The csv itself has the CRC32 value of 017C2744 to be original. The trigger name is ERRARDVD35 and it belongs to the group Fine Art Erotica, with sub grouping of Errotica Archives. The type for the csv file is regular (there are other types, like asian, wsc and such).

To use a csv file, you must have a program. Such programs are called Collection Managers. The earliest useful collection manager is ScanSort, followed by Hunter and PicCheck. The current tools of choice are PServeCheck or PSProVerify. These two latter tools can be hard to come by, but know a friend who knows somebody, who's connected and may know where to ask for directions OR ask on your favorite csv site *hint*.

Saturday, September 12, 2009

Effective use of PSProVerify - burning collections



If you haven't read the article below about Intelli Burn, the collection burning feature for CD's, DVD's and other types of media. Then go read it. This is the tips for that article.

Todays tip is for the burning feature and the tip states that it is rather useful if you:

  • Mix collections of different sizes
  • Keep it to a minimum of groups

If you mix your burns with a mix of DVD and CD sized collections then the algorithm can better choose 'filler' collections to better utilize the space on the DVD's.

If you have a low count of groups, then it is much easier for the algorithm to mix the different collections as there is lesser constraints.

Following these two tips will give you the best utilization of the DVD's or whatever media you burn to.

Thursday, September 10, 2009

Intelli Burn

This article is about PSPV (PSProVerify) intelligent way of burning collections to any type of media. Look at the screenshot below and notice that some collections has been picked. You rightclick and choose 'Burn'


Choosing 'Burn' will take you a new dialog where you can group your collections, on this picture you can set a fill limit. This is a limit which says how much space on the DVD (as example) you want filled before you will burn it. 0% means that you do not care if you burn a DVD with just 1 byte. Choosing 70% means that atleast 70% of the free space on a DVD will be occupied by data. Once you have grouped your collections and set your fill limit percentage, you should 'Get ready to burn'


'Get ready to burn' will take you to the second tab and you can now see how filled the DVD's will be if the suggested burning will take place. Notice the red circles. Click 'Burn select Discs' to proceed with the actual burning


Let's set the fill limit to 70% and the do the process over again. Notice the change of which will be burned.


You should be able to that the grouped DVD with a fill percentage of 66.17% is NOT among the things which will actually be physically burned to DVD's.

One of the intelligent things which the burning feature in PSProVerify has is the selection of what is packed on each DVD. PSPV utilizes a algorithm called backpacking which finds the best combination of which collections to store to utilize as much freespace on the DVD or other media type, as possible. Very smart, and you use a minimum of DVD's to burn a maximum of collections.

Do not choose a large number og collections and do not set a fill-limit with a too high percentage. The algorithm can take extremely long time to finish finding the proper mix of collections to store on each piece of media.

Wednesday, September 2, 2009

Hunter Superior - new collection management tools - comparisons

When it comes to verifying and sorting your collections against the csvs then you only have a few tools to choose from.

  • ScanSort
  • The!Checker / PicCheck
  • Hunter
  • PServeCheck also called PSC or  PSCU
  • PSProVerify also called PSPV or PSClite

ScanSort is a commandline tools which works on both the Unix/Linux platform aswell as the Windows. ScanSort dates back to 1999 and the source code public domain and can be download alof of places. The lastest version is considered to be 1.81-2.

The!Checker is a Windows based tool which originates back to 2006. The!Checker or PicCheck which is a common name for the tool must be considered outdated as larger size csvs are more common. If you get nostalgic then you can always have a look here: http://freenet-homepage.de/vhorch/piccheck.htm. PicCheck is considered to be simple and quite easy to learn.

Hunter dates back to 1998-1999 and is also a Windows based application, although rumors of a Linux version exists. Hunter has far more features than ScanSort and PicCheck. Hunter is still used today. You can get it here: http://broquard-ebooks.com/hunter/

PServeCheck is a native Windows based tool which is still maintained today (2009). PSC or PSCU which are common used names for the tool is full featured with a load of very integrated and easy to use functions and methods. PSCU is designed to be easy and yet to be able to handle several thousand different collections. One of the great steps forward with PSCU compared to ScanSort/The!Checker and Hunter is the ability to automatically get updated csvs once they are available. Further more it is able to automatically check your collections and automatically download files which match the csvs you use. PSCU also incorporates csvs with states such as Ongoing, Pre-Final, Final and Re-Burn. It integrates to Nero or ImgBurn and handles drive arrays beyond 10Tb. PSC comes as an integrated part of PhotoServe, but can also work alone. PSCU uses around 50mb memory and is written in C++.

PSProVerify is quite similar to PServeCheck and works on the Windows platform. PSPV is also still maintained (2009). PSPV, which is the common name for the tool is slightly different PSCU as the graphical user interface doesn't look like normal Windows. PSPV uses the Tk GUI for Perl. PSPV it not 100% bound to Windows and should be portable to other platforms, but is not planned to be done by the author (since it is open source, you can do it if you want).  PSPV has a very rich set of features and is extremely integrated and it is designed to set it up once and let it run fully automated, it is not designed for doing everything manualy, although you can. PSPV is known to be fully automatic and draws on several channels for downloading. Especially the very well integrated bittorent integration is appreciated. With PSPV you get almost all of the features of PSCU plus many automated tools for maintaining large collections. PSPV uses around 90->260mb memory and is written in Perl.

The list below may be subject to change as new versions and updates appear. 

Feature
PSCU PSPV
Description
Automatic CSV download
x
x
You can "subscribe" to several different websites and the tool will automatically download and check the collections you've set to automatically monitor for new csvs.
Automatic collection checking
x
x
Automatically checking of collections based on downloads and changes in csv state or count
Automatic hunting for new csvs and files
x
x
Ability to hunt for csvs and files to match csvs
Automatic download of collection files
x
x
The tools can be set to automatically leeching files if the integration with PS is in place.
HDD space management systems
x
x
PSCU has Simple Collection Management. PSPV has Advanced Collection Management.
Hunting inside RAR and ZIP files
x
x
Ability to unzip/unrar (single level)
Creation of need / miss_wrong csvs
x
x
If not all files in a csv are present then it is possible to create a new csv which holds the file information of the files needed
Fill of need / miss_wrong csvs
x
x
Ability to fill a needs csv. Both have a possibility to mass fill a bunch of need csvs in one run.
Burn to media CD / DVD etc
x
x
Nero and ImgBurn integration, even ISO support
Configurable auto hunt features
x
x
Ability to configure which collections use which folders for autohunt runs. Including 'removed' folders.
Dupe distribitor
x
Ability to fill csvs with files existing in other csvs. Saves leech time, if external dupes are present.
Custom collection and triggers
x
x
Ability to handle your own personal non-official csvs and groups. PSPV has integrated creation of new groups, whereas PSCU needs manual file (.TRG) creation. Both have ability to create new custom triggers in the custom .TRG files.
Official trigger and csv support to (El-Toro / VG)
x
x
Yes, currently ET is offline and only the VG repo array is supported.
CSV/Hunt cache
x
x
Use of cache files for faster checking and hunting.
CSV creation+upload
x
x
Ability for csv maintainers to to create csvs. Upload to the repo array is supported.
Virtual groups
x
Ability to create your own groups and arrange/merge official and custom collections into your own layout and naming
Location / drive view
x
x
Ability to view collections by drives.
Priority on downloads
x
x
Ability to set priority of which is more important. PSCU's feature is rather intuitive, whereas PSPV's could lack some easyness. PSPV's is highly configurable though, and supports more than 6 different priority levels
Bittorent support
(x)
x
PSCU's bittorent support is a somewhat manual experience, whereas PSPV's is highly automatic. Both tools integrate with uTorrent as client.
Find / search feature
x
(x)
Ability to search for triggers and inside csv files. PSCU's is a simple Ctrl+F and automatic wildcard supported. PSPV's needs a few settings to perform similar.
HTML/Excel/XML and csv reports and view exports
x
x
PSCU can export to all types. PSPV mainly to tabulator seperated lists.
Auto setup series
x
x
Configure the tool to automatically setup new collections as csvs appear in channels (if PS is integrated) or present on the repo array
Integrated with PS
x
x
Yes
Support for offline / backuped collections
x
x
Yes
Simulate HDD space management
x
Only PSPV can run a simulated run of the ACM before doing it.
Removal of orphaned csvs / folders
x
Only PSPV can clean orphaned csvs / folders.
Collection recycler
x
PSPV only, rather dangerous. But the feature is used to accumulate a bunch of CD collections into a DVD collection, as example.
Clean up of torrents
x
PSPV only.
Disk verifier
x
PSPV only.
Delete to recycled bin
(x)
x
PSCU deletes to the removed folder by default. Removed folder is a special folder where stuff present in a collection directory is moved to if it does not belong there according to the csv.
Auto update/leech based on stats value
x
PSPV can automatically turn on leeching / updating if a value for a collection is greater than a custom set pre-defined value. Based og the Stats bot. Only for use with PS.
Filters / views
x
x
PSCU = Leeches, Virtual Groups, New CSVs, Maintained CSVs, Marked collections, Auto get CSV collections, Auto update collections, Badly setup collections, Suspended collections, No default setup collections, Ready to burn, Burned, Not setup collected series, All setup collections, Offline collections, All collections, Complete, Incomplete, Ongoing, Pre-Final, Final, Reburn, Favorites, Hidden, Location view, Leech history, Orphaned collections, Priority, Bittorent Seed, Bittorent Locked, Collection hunt and Need check.

PSPV = All available collections, ACM auto purge list, Advanced autohunt set, All setup collections, Bad setup, Better CSV existing, Bittorent active, Burned collections, CSV-change history, Discontinued series, Incomplete collections, Leech history, Self maintained collections, Ready to burn, Send history, Custom Filters, and Drives.
Suspend collections
x
Ability to temporarely turnoff a bunch of collections if you detach an external drive or something similar.
Collection size bar
x
x
Shows color schemes based on common media sizes, CD, DVD etc.
Hide offline, burned etc collections by default
x
x
Both tools has ability to filter on / off based on special collection states.
View csv, report etc
x
x
Simple integration to default viewers of reports (.txt) files and .csv files. PSPV does not use the default Windows program for some reason.
View with ACDSee
x
PSCU only.
Explore removed, autorhunt etc folders
x
x
Starts Explorer in the specific folders.
Smart cache
x
Greately enhances speed when checking. For use with PS only.
Restore previous GUI settings
x
(x)
PSCU is able to open the program in the same state as you left it. Columns, tree structure, folde in/out. Everything stays. PSPV opens the last opened trigger/collection group.
Estimated times and speed
x
PSCU displays estimated times on checking and speed on csv creation.
Tray integration, label change and password protect
x
PSCU minimizes to tray, you can change the application name to something else and password protect the program so nobody but you can open it.
Redistribute and alter column sequence
x
(x)
PSCU's works by drag'n'drop. PSPV's needs to edit a list of possible columns and then moved them up/down (left/right) to change sequence.
Sort by column
x
(x)
PSCU can sort both ascending and descending. PSPV only supports ascending sort.
Configurable timer intervals between auto checks
x
PSCU has internal defined timer settings for automatic checking etc. PSPV's can be configured by the user.
Bundles
x
PSPV supports the creation and handle of bundles if sizes exceed a user defined limit. Great if you have ALOT of stuff.
Count indicators
x
PSPV shows a numeric indicator (like Outlook) on how many triggers/collections are setup in a group. Color is also used.
Custom filters
x
PSPV lets you customize filters based on pre-defined filters. Really great for the skilled person.


The issues (good/bad) can be cut down to the following:

PServeCheck
  • Fastest tool, both checking and making and hunting.
  • Nice GUI.
  • Simple and many useful pre-defined filters.
  • Works out-of-the-box.
  • No special skills needed.
  • Lacks bittorent integration compared to PSPV.
  • Has Virtual Groups.
  • Un-updated help file.
PSProVerify
  • Most configurable tool.
  • Ability to work like a robot, no human intervention needed. Can be very automatic.
  • Custom filters. A very powerful feature. 
  • Regular expressions (RegEx) skills to a degree are needed.
  • Value based leech is a super feature.
  • Advanced Collection Management for HDD space.
  • Great bittorent support.
  • Needs a bunch of configurering.
  • No help file.


I have heard of people using both tools depending on their task at hand. If you were to go on holiday for 3 weeks, then you can set PSPV to handle things like you were at home and did everything yourself. It will require quite a bit of setup and understanding, but absolutely possible. If you don't leave your computer online for 3 weeks, then PSCU may be the choice. If you're for the easy, fast and simple then the name is PSCU, if you like configurable, tuneable, complex then go for PSPV. Or mix them. But DO NOT HAVE THEM BOTH RUNNING ON THE SAME TIME.

Here's a few screenshots.


PSProVerify









PServeCheck