'ZenCLSH - Zentastic Craigslist Search Harvester 'A tool for automatically downloading search results from Craigslist using multiple search configurations. 'Written by Shannon Larratt ' snowrail@gmail.com ' http://www.zentastic.com/ 'Software homepage is at http://www.zentastic.com/blog/zenclsh/ 'ZenCLSH is free software and is provided with no warranty or anything like that. 'This tool is not endorsed or otherwise affiliated with Craigslist or Kijiji or anyone else. 'Use at your own risk. 'Notes about source code: 'Copyright (c) 2008-2009 Shannon Larratt. 'All rights reserved. You may use portions of this code in your own projects (credit is appreciated but 'not required), but you may not release your own versions of this program without getting my consent first. 'Compiled under PowerBasic PB/Win 9.02 available from http://www.powerbasic.com/ #COMPILE EXE 'Well duh #DIM ALL 'No being lazy #OPTIMIZE SPEED 'Who optimizes for size any more? #TOOLS OFF 'No debug/trace tools #RESOURCE "zenclsh.pbr" 'This just contains the program icon 'Version history ' 1.00 Jul 26 2008 - Original Release ' 1.01 Jul 28 2008 - Fixed bug that was forcing "zenclsh.html" filename ' - Checks for updates at zentastic.com/blog/zenclsh/ ' - If new entries are in place, but file has not been saved, ask before close ' - Clear options on list (delete all, kill all entries two months old or more) ' - clcities.txt and clsearch.txt are used as default, but templatename.cit and .trm are used if they exist ' - name of last search is saved to zenclsh.ini ' 1.02 Jul 29 2008 - Added support for kijiji searches ' - Fixed bug where modified city lists weren't being respected ' 1.03 Aug 04 2008 - Adds self to start menu if requested ("startmenu" line in zenclsh.ini) ' - Resizing of window now permitted, with size saved to setup file ' - Pretty icon, ooooh! ' - Fixed bug where blank lines were getting added ' 1.04 Aug 06 2008 - Converted file selector to dropdown combobox for managing multiple searches ' 1.05 Aug 06 2008 - Minor Bugfix: doesn't allow load/reload if there are unsaved items ' --NEW--VERSION-- - -------------------------------------------------------------------------------- ' 2.00 Nov 19 2009 - Complete rewrite of the original ZenCLSH! Woo woo! However, some features were lost in the process (temporarily) until they ' can be recoded ' 2.01 Nov 19 2009 - Minor bugfix: Deep scan threads colliding fixed ' 2.02 Nov 20 2009 - Slightly changed HTML output, including tagging the day's downloads with "NEW" ' - When saving CSV files and HTML files, you can output everything, or just today's new entries ' - Permissions-level bug in "view in browser" shell command should be fixed now [untested fix] ' - Collisions in listview no longer happen ' - You can select multiple items from the results listview to output or view just those items ' - Offers to add itself to start menu ' - Added "About" dialog and moved the "visit website" button there ' - Can now specify that search should be of titles only, not body text ' - Can now specify that results should only include ads with pictures ' - Clicking on column headers sorts list by tha column ' - Source code made public... I sure hope it's not too embarassing ' - Search name can be specified on the command line -- ie. "ZenCLSH2 searchname" 'To-Do List... and To-Consider List ' * Activate Kijiji searching. ' * Mapping between Kijiji and Craigslist sections and cities, if possible. ' * View HTML sometimes has HTML problems where unclosed tags cause problems. ' * Deep Scan of expired postings don't record correctly (include headers, etc). ' * Filter results list via some sort of search. ' * HTML output should be prettier, and should be split up by date as well -- should use a template. ' * Hover over result pops up a preview of the full text if it's known... if I can figure out how to detect a hover! ' * Give a redundancy warning if "Car & Trucks - By Dealer" and "Car & Trucks - By Dealer" are both selected (since "Cars+Trucks" does both sections), or if one is select ' at the same time as that parent category. Same goes for any others ("Real Estate For Sale/By Broker/By Owner" probably, not sure if there are others). ' * Make diagnostic page a little prettier and have more info (# search results, # search matches, bytes downloaded, etc.). ' * ZenCLSH settings page should warn if tab is changed without applying or cancelling settings. ' * Search name could be bumped down into search results tab. ' * Progress bar should work for deep scan also. ' * Figure out why section/city csv files sometimes erroneously show as not found (might be open in thread?). ' * See if I can figure out why the scan is 100% functional at first, and then starts failing a little way in (ie. overloading router, ISP, OS, or a PB issue... ' if it's a PB issue, fix it). ' * Add more command line options so that, for example, the tool can be run as a scheduled task. ' * About dialog should be forced to the front and stay there until closed. 'Includes %USEMACROS = 1 $INCLUDE ONCE "WIN32API.INC" $INCLUDE ONCE "COMMCTRL.INC" $INCLUDE ONCE "COMDLG32.INC" $INCLUDE ONCE "PBForms.INC" 'Globals... Yeah, I know I use a lot of them GLOBAL connectFailUp AS DWORD GLOBAL tabSearchResults AS DWORD GLOBAL tabSearchSettings AS DWORD GLOBAL tabSoftwareSettings AS DWORD GLOBAL tabDiagnostics AS DWORD GLOBAL mainDialogHandle AS DWORD GLOBAL aboutDialogHandle AS DWORD GLOBAL dlWindowHandle AS DWORD GLOBAL startmenu$ GLOBAL newX AS LONG, newY AS LONG GLOBAL gCS AS CRITICAL_SECTION 'critical section structure to keep listview collisions from happening GLOBAL doCityList$ GLOBAL doTermList$ GLOBAL doSectionList$ GLOBAL hThread???, vThread???, dlThread???, cThread???, lThread??? GLOBAL sThread???(), dThread???(), dActive() AS LONG GLOBAL sTerm$(), sCity$(), sSection$(), sPrettyCity$(), sPrettySection$() GLOBAL lockListview AS LONG GLOBAL SearchRunning AS LONG GLOBAL DeepScanRunning AS LONG GLOBAL SearchAbort AS LONG GLOBAL DeepScanAbort AS LONG GLOBAL UnsavedResults AS LONG GLOBAL CurrentSearchname$, forceSearchName$ GLOBAL SearchCraigslist AS LONG GLOBAL SearchKijiji AS LONG GLOBAL AutoDeepScan AS LONG GLOBAL CheckForUpdates AS LONG GLOBAL DiscardSearches AS LONG GLOBAL searchTitles AS LONG GLOBAL hasPictures AS LONG GLOBAL MaxResults AS LONG GLOBAL MaxThreads AS LONG GLOBAL MaxThreads2 AS LONG GLOBAL thisSearchName$ GLOBAL thisTermList$ GLOBAL UserAgent$ $WEBPAGESITE = "www.zentastic.com" 'My site $WEBPAGEPAGE = "/blog/zenclsh/" 'Location of software page on my site $PUT_OK = "qwertyuiopasdfghjklzxcvbnmQWERTYUIOPASDFGHJKLZXCVBNM1234567890.- " %VERSION = 202 'Current version %HH_DISPLAY_TEXT_POPUP = &HE GLOBAL SettingsWindowOpen AS LONG, CityWindowOpen AS LONG, SectionWindowOpen AS LONG, TermWindowOpen AS LONG GLOBAL CityList$() GLOBAL NumCities AS LONG GLOBAL RegionArray$(), CountryArray$(), CityArray$() 'These three arrays (and the next two lines) maintain handle data for the treeview GLOBAL RegionArrayHandle() AS DWORD, CountryArrayHandle() AS DWORD, CityArrayHandle() AS DWORD GLOBAL RegionArrayStatus() AS LONG, CountryArrayStatus() AS LONG, CityArrayStatus() AS LONG GLOBAL tempNumRegions AS LONG, tempNumCountries AS LONG, tempHandle AS DWORD GLOBAL SectionList$() GLOBAL NumSections AS LONG GLOBAL SectionArrayHandle() AS DWORD GLOBAL ResultsArray$() GLOBAL SelectedItems() AS LONG GLOBAL numResults AS LONG TYPE HH_POPUP cbStruct AS LONG hinst AS LONG idString AS LONG pszText AS LONG pt AS POINTAPI clrForeground AS LONG clrBackground AS LONG rcMargins AS RECT pszFont AS LONG END TYPE 'Constants %IDD_ZENCLSHDIALOG = 101 '1-series is the main dialog %IDC_SYSTAB = 1001 %IDC_SEARCHNAME = 1003 %IDC_LOADSEARCH = 1005 %IDC_SAVESEARCH = 1006 %IDC_PROGRESS = 1007 %IDC_CLEARRESULTS = 1010 %IDC_OUTPUT_CSV = 1011 %IDC_VIEW_HTML = 1012 %IDC_DEEPSCAN = 1014 %IDC_GO_SEARCH = 1013 %IDC_RESULTSLIST = 1008 %IDC_CITIESTREE = 1016 %IDC_CITIESALL = 1017 %IDC_CITIES_CLEARALL = 1018 %IDC_SECTIONS_TREE = 1020 %IDC_SECTIONSELECTALL = 1021 %IDC_SECTION_CLEARALL = 1022 %IDC_LABEL = 1023 %IDC_SEARCHTERMS_BOX = 1024 %IDC_SEARCHNAME_BOX = 1004 %IDC_CHECK_CL = 1025 %IDC_CHECK_KIJIJI = 1026 %IDC_CHECK_AUTODEEP = 1027 %IDC_CHECKZEN = 1028 %IDC_DISCARD = 1032 %IDC_SETTINGS_CANCEL = 1030 %IDC_APPLY_SETTINGS = 1029 %IDC_VISITWEB = 1031 %IDC_MAXLABEL = 1033 %IDC_MAXRESULTS_BOX = 1034 %IDC_THREADSLABEL = 1035 %IDC_THREADSRESULTS_BOX = 1036 %IDC_THREADS2LABEL = 1037 %IDC_THREADS2RESULTS_BOX = 1038 %IDC_USERAGENT = 1039 %IDC_USERAGENT_BOX = 1040 %IDC_STHREAD = 1050 'This ID applies to all 12, just add the # to get the real ID %IDC_DTHREAD = 1070 'This ID applies to all 12, just add the # to get the real ID %IDC_CHECK_SEARCHTITLES = 1100 %IDC_CHECK_HASPICTURES = 1101 %IDD_DOWNLOAD = 201 '2-series is the section/city download dialog %IDC_OUTPUTBOX = 2001 %IDC_BEGIN_DOWNLOAD = 2002 %IDD_ABOUTDIALOG = 301 '3-series is the about dialog %IDC_LABEL1 = 3001 %IDC_LABEL2 = 3002 %IDC_LABEL3 = 3003 %IDC_LABEL4 = 3004 %IDC_LABEL5 = 3005 %IDC_LABEL6 = 3006 %IDC_LABEL7 = 3007 %IDC_LABEL8 = 3008 %IDC_ABOUT_VISITWEBSITE = 3010 %IDC_ABOUT_THATSCOOL = 3009 %IDC_ABOUT_ICON = 3011 'Declarations (local - my code) DECLARE SUB OutputHTML(fileOut$, outputMode AS LONG) 'spits out html file (outputMode = 1 for all, 2 for new only, 3 for selected items in selectedItems()) DECLARE FUNCTION CCtoRegion$(CountryCode$) DECLARE FUNCTION GetCountry$(CuntNo AS LONG) 'Return the alpha-2 country code and full name DECLARE FUNCTION numFromMonth(numName$) AS LONG 'convert text month to numeric DECLARE CALLBACK FUNCTION ShowZENCLSHDIALOGProc() DECLARE FUNCTION ShowZENCLSHDIALOG(BYVAL hParent AS DWORD) AS LONG DECLARE CALLBACK FUNCTION ShowDOWNLOADProc() DECLARE FUNCTION ShowDOWNLOAD(BYVAL hParent AS DWORD) AS LONG DECLARE FUNCTION ToPut$(PutIn$) AS STRING 'convert string to PUT-able string DECLARE SUB WriteSettings() DECLARE SUB LoadSettings() DECLARE FUNCTION CheckVer(BYVAL dummy AS DWORD) AS DWORD DECLARE FUNCTION httpGrab$(site$, file$, allowredirect AS LONG) 'Simply grabs a file via HTTP GET DECLARE FUNCTION Between$(inString$, starter$, finisher$) 'find the text between two markers DECLARE FUNCTION GetCraigsRSS(Query$, City$, Section$, Item$(), Link$(), itemDate$(), Description$()) AS LONG 'Harvest a Craigslist RSS feed DECLARE FUNCTION doLaunchDeep(BYVAL dummy AS DWORD) AS DWORD 'This function just launches a deep scan DECLARE SUB ShowPopupHelp(sText$) 'Pops up a help window for the context-sensitive help system DECLARE FUNCTION CSVsafe$(inString$) 'Makes sure a string is safe to use in a CSV file DECLARE FUNCTION doDeepScan(BYVAL whichScan AS DWORD) AS DWORD 'Execute a deep scan (assumes list is pre-saturated) DECLARE SUB doCities() DECLARE SUB WriteSettings() 'Save the settings file DECLARE SUB LoadSearch(noWarn AS LONG) 'load search results from disk DECLARE SUB SaveSearch() 'Save current search results DECLARE SUB LoadSettings() 'Load settings file DECLARE FUNCTION doDownload((BYVAL dummy AS DWORD) AS DWORD 'Download city/section settings (threaded to avoid crashes) DECLARE FUNCTION ParseCSV$(inString$, termNo AS LONG) 'Pull out a string from a CSV and reverse the encoding DECLARE CALLBACK FUNCTION ShowABOUTDIALOGProc() DECLARE FUNCTION ShowABOUTDIALOG(BYVAL hParent AS DWORD) AS LONG DECLARE FUNCTION realEXEpath$() 'Ensure there's a trailing slash... for some reason it seems inconsisten 'Declarations (local - other people's code) DECLARE FUNCTION createshortcut(csidl AS DWORD, link AS STRING, SOURCE AS STRING, workdir AS STRING, scomment AS STRING) AS LONG DECLARE FUNCTION specialfolder(pidl AS DWORD) AS STRING 'Declarations (Prototypes) DECLARE FUNCTION ishelllink_call0( BYVAL punk AS LONG ) AS LONG DECLARE FUNCTION ishelllink_call1( BYVAL punk AS LONG, BYVAL p1 AS LONG ) AS LONG DECLARE FUNCTION ishelllink_call2( BYVAL punk AS LONG, BYVAL p1 AS LONG, BYVAL p2 AS LONG ) AS LONG 'Declarations (external) DECLARE FUNCTION HtmlHelp LIB "hhctrl.ocx" ALIAS "HtmlHelpA" (BYVAL hwndCaller AS LONG, pszFile AS ASCIIZ, BYVAL uCommand AS LONG, BYVAL dwData AS LONG ) AS LONG 'EXECUTION OF CODE STARTS HERE FUNCTION PBMAIN() LOCAL res AS LONG InitializeCriticalSection gCS 'This program has critical sections PBFormsInitComCtls (%ICC_WIN95_CLASSES OR %ICC_DATE_CLASSES OR %ICC_INTERNET_CLASSES) 'Who knows what this does... forceSearchName$ = COMMAND$(1) 'Command line option -- ie. ZenCLSH2 searchname IF forceSearchName$ <> "" THEN 'See if the search name is valid IF LCASE$(RIGHT$(forceSearchName$, 4)) = ".zen" THEN 'extension has been specified -- oops, it's not required forceSearchName$ = LEFT$(forceSearchName$, LEN(forceSearchName$) - 4) 'remove extension END IF IF DIR$(forceSearchName$ & ".zen") = "" AND DIR$(realEXEpath$ & forceSearchName$ & ".zen") = "" THEN 'file not found MSGBOX "Specified search """ & forceSearchName$ & """ can't be found. Initializing it as a new search.", %MB_SYSTEMMODAL OR %MB_ICONWARNING, "Warning" END IF END IF 'Search thread arrays DIM sThread???(1 TO 12) DIM dThread???(1 TO 12) DIM dActive(1 TO 12) DIM sTerm$(1 TO 12) DIM sCity$(1 TO 12) DIM sSection$(1 TO 12) DIM sPrettyCity$(1 TO 12) DIM sPrettySection$(1 TO 12) 'Load settings (or set up defaults) and cities/sections LoadSettings() IF numSections <> 0 AND numCities <> 0 THEN IF startMenu$ <> "done" THEN 'Offer to add ZenCLSH to start menu if it hasn't been done (or explicitly not done) already res = MSGBOX("Would you like to add ZenCLSH to your start menu? Select 'yes' to do so, 'no' not to, or 'cancel' to ask again later.", %MB_YESNOCANCEL + %MB_ICONQUESTION + %MB_APPLMODAL, "Start Menu?") IF res = %IDYES THEN 'Yes, add it createshortcut %CSIDL_PROGRAMS, "ZenCLSH.lnk", realEXEpath$() & "ZenCLSH2.exe", realEXEpath$(), "Run the Zentastic Craigslist Search Harvester" startMenu$ = "done" ELSEIF res = %IDNO THEN 'Nope, don't add it startMenu$ = "done" ELSE 'Maybe later startMenu$ = "later" END IF END IF 'Set up results array, which contains all the results of this search to date DIM ResultsArray$(1 TO MaxResults, 1 TO 7) '1=Date, 2=Title, 3=City, 4=Section, 5=URI, 6=FullText, 7=NewThisSession DIM SelectedItems(1 TO MaxResults) AS LONG numResults = 0 'Start program by creating main window ShowZENCLSHDIALOG %HWND_DESKTOP ELSE 'Since we don't have all the cities/sections, do that now ShowDOWNLOAD %HWND_DESKTOP END IF DeleteCriticalSection gCS 'Kill off critical section handler END FUNCTION SUB WriteSettings() 'Save the settings file LOCAL IniFile$ IniFile$ = realEXEpath$() & "zenclsh.ini" LOCAL ff AS LONG ff = FREEFILE OPEN IniFile$ FOR OUTPUT AS #ff PRINT #ff, "CurrentSearchName: " & CurrentSearchname$ PRINT #ff, "SearchCraigslist: " & FORMAT$(SearchCraigslist) PRINT #ff, "SearchKijiji: " & FORMAT$(SearchKijiji) PRINT #ff, "AutoDeepScan: " & FORMAT$(AutoDeepScan) PRINT #ff, "CheckForUpdates: " & FORMAT$(CheckForUpdates) PRINT #ff, "DiscardSearches: " & FORMAT$(DiscardSearches) PRINT #ff, "MaxResults: " & FORMAT$(MaxResults) PRINT #ff, "MaxThreads: " & FORMAT$(MaxThreads) PRINT #ff, "MaxThreads2: " & FORMAT$(MaxThreads2) PRINT #ff, "Width: " & FORMAT$(newX) PRINT #ff, "Height: " & FORMAT$(newY) PRINT #ff, "UserAgent: " & UserAgent$ PRINT #ff, "StartMenu: " & startmenu$ PRINT #ff, "SearchTitles: " & FORMAT$(searchTitles) PRINT #ff, "HasPictures: " & FORMAT$(hasPictures) CLOSE #ff END SUB SUB LoadSearch(noWarn AS LONG) 'load search results from disk LOCAL SearchFile$, tempCityList$, tempSectionList$, tempSearchTerms$, thisLine$ LOCAL oldestSearch$ LOCAL thisYear AS LONG, thisMonth AS LONG, thisDay AS LONG LOCAL ff AS LONG, ctr AS LONG LOCAL thisParent AS DWORD, thisParent2 AS DWORD 'Grab the name of the search and reduce it appropriately CONTROL GET TEXT mainDialogHandle, %IDC_SEARCHNAME_BOX TO CurrentSearchname$ REPLACE ANY CHR$(".*?/\""") WITH " " IN CurrentSearchname$ REPLACE " " WITH "" IN CurrentSearchname$ CONTROL SET TEXT mainDialogHandle, %IDC_SEARCHNAME_BOX, CurrentSearchname$ 'Just in case it's changed 'Does the search file exist? IF INSTR(CurrentSearchname$, "\") = 0 THEN SearchFile$ = realEXEpath$() & CurrentSearchname$ & ".zen" 'Add local path if we don't have it already ELSE SearchFile$ = CurrentSearchname$ & ".zen" END IF IF ISFILE(SearchFile$) = %FALSE THEN IF noWarn = %FALSE THEN MSGBOX """" & CurrentSearchname$ & """ can't be found. Perhaps it is a new search?", %MB_SYSTEMMODAL OR %MB_ICONWARNING, "Warning" END IF ELSE ff = FREEFILE OPEN SearchFile$ FOR INPUT AS #ff LINE INPUT #ff, CurrentSearchname$ LINE INPUT #ff, tempCityList$ LINE INPUT #ff, tempSectionList$ LINE INPUT #ff, tempSearchTerms$ 'Now load the search results (which should also be written to the listview) numResults = 0 IF DiscardSearches = %FALSE THEN oldestSearch$ = "0000-00-00" ELSE thisYear = VAL(RIGHT$(DATE$, 4)) thisMonth = VAL(LEFT$(DATE$, 2)) thisDay = VAL(MID$(DATE$, 4, 2)) DECR thisMonth IF thisMonth = 0 THEN thisMonth = 12 DECR thisYear END IF DECR thisMonth IF thisMonth = 0 THEN thisMonth = 12 DECR thisYear END IF oldestSearch$ = FORMAT$(thisYear, "0000") & "-" & FORMAT$(thisMonth, "00") & "-" & FORMAT$(thisDay, "00") END IF LISTVIEW RESET tabSearchResults, %IDC_RESULTSLIST 'erase listview of results DO UNTIL EOF(ff) LINE INPUT #ff, thisLine$ 'Grab encoded results line from file INCR numResults 'Increment counter IF numResults > maxResults THEN DECR numResults MSGBOX "The results list has overflowed (" & FORMAT$(numResults, "0,") & " elements). Please increase max results in the settings.", %MB_SYSTEMMODAL OR %MB_ICONERROR, "Error" EXIT DO END IF FOR ctr = 1 TO 6 ResultsArray$(numResults, ctr) = ParseCSV$(thisLine$, ctr) 'Pull out individual fields NEXT ResultsArray$(numResults, 7) = "" 'Not new this session IF ResultsArray$(numResults, 2) = "" THEN DECR numResults ITERATE DO END IF IF ResultsArray$(numResults, 1) < oldestSearch$ THEN 'Search result is outdated and should be thrown away DECR numResults ELSE LISTVIEW INSERT ITEM tabSearchResults, %IDC_RESULTSLIST, numResults, 0, ResultsArray$(numResults, 1) 'Set date column LISTVIEW SET TEXT tabSearchResults, %IDC_RESULTSLIST, numResults, 2, ResultsArray$(numResults, 2) 'Set title colum LISTVIEW SET TEXT tabSearchResults, %IDC_RESULTSLIST, numResults, 3, ResultsArray$(numResults, 3) 'Set city colum LISTVIEW SET TEXT tabSearchResults, %IDC_RESULTSLIST, numResults, 4, ResultsArray$(numResults, 4) 'Set section colum IF LEN(ResultsArray$(numResults, 6)) > 50 THEN 'We have downloaded the full text LISTVIEW SET TEXT tabSearchResults, %IDC_RESULTSLIST, numResults, 5, "Yes" 'Set full text colum END IF LISTVIEW SET USER tabSearchResults, %IDC_RESULTSLIST, numResults, numResults 'Set user data to pointer to full info in ResultsArray$() END IF LOOP CLOSE #ff 'Set terms... that's easy, just set it! CONTROL SET TEXT tabSearchSettings, %IDC_SEARCHTERMS_BOX, tempSearchTerms$ 'Set up the cities by going through them one by one and seeing if they match 'Note: this really needs to flip the parents also... tempCityList$ = "," & tempCityList$ & "," 'This makes the searching very easy FOR ctr = 1 TO numCities IF INSTR(tempCityList$, "," & CityList$(ctr, 5) & ",") > 0 THEN CityList$(ctr, 6) = FORMAT$(ABS(%TRUE)) 'Turn on checkbox TREEVIEW SET CHECK tabSearchSettings, %IDC_CITIESTREE, CityArrayHandle(ctr), %TRUE CityArrayStatus(ctr) = %TRUE 'Expand branch so we can see all visible options TREEVIEW GET PARENT tabSearchSettings, %IDC_CITIESTREE, CityArrayHandle(ctr) TO thisParent TREEVIEW SET EXPANDED tabSearchSettings, %IDC_CITIESTREE, thisParent, %TRUE TREEVIEW GET PARENT tabSearchSettings, %IDC_CITIESTREE, thisParent TO thisParent2 TREEVIEW SET EXPANDED tabSearchSettings, %IDC_CITIESTREE, thisParent2, %TRUE ELSE CityList$(ctr, 6) = FORMAT$(ABS(%FALSE)) TREEVIEW SET CHECK tabSearchSettings, %IDC_CITIESTREE, CityArrayHandle(ctr), %FALSE CityArrayStatus(ctr) = %FALSE END IF NEXT 'Set up the regions by going through them one by one and seeing if they match tempSectionList$ = "," & tempSectionList$ & "," 'This makes the searching very easy FOR ctr = 1 TO numSections IF INSTR(tempSectionList$, "," & SectionList$(ctr, 2) & ",") > 0 THEN SectionList$(ctr, 4) = FORMAT$(ABS(%TRUE)) 'Turn on checkbox TREEVIEW SET CHECK tabSearchSettings, %IDC_SECTIONS_TREE, SectionArrayHandle(ctr), %TRUE 'Expand branch so we can see all visible options TREEVIEW GET PARENT tabSearchSettings, %IDC_SECTIONS_TREE, SectionArrayHandle(ctr) TO thisParent IF thisParent > 0 THEN TREEVIEW SET EXPANDED tabSearchSettings, %IDC_SECTIONS_TREE, thisParent, %TRUE END IF ELSE SectionList$(ctr, 4) = FORMAT$(ABS(%FALSE)) TREEVIEW SET CHECK tabSearchSettings, %IDC_SECTIONS_TREE, SectionArrayHandle(ctr), %FALSE END IF NEXT END IF END SUB SUB SaveSearch() 'Save current search results LOCAL SearchFile$, tempCityList$, tempSectionList$, tempSearchTerms$ LOCAL thisCheck AS LONG, ctr AS LONG LOCAL ff AS LONG 'Grab the name of the search and reduce it appropriately CONTROL GET TEXT mainDialogHandle, %IDC_SEARCHNAME_BOX TO CurrentSearchname$ REPLACE ANY CHR$(".*?/\""") WITH " " IN CurrentSearchname$ REPLACE " " WITH "" IN CurrentSearchname$ CONTROL SET TEXT mainDialogHandle, %IDC_SEARCHNAME_BOX, CurrentSearchname$ 'Just in case it's changed 'Grab cities tempCityList$ = "" FOR ctr = 1 TO numCities 'Go through all the cities IF CityArrayStatus(ctr) <> %FALSE THEN IF tempCityList$ <> "" THEN tempCityList$ = tempCityList$ & "," tempCityList$ = tempCityList$ & CityList$(ctr, 5) 'Add search URL to our to-do list END IF NEXT 'Grab sections tempSectionList$ = "" FOR ctr = 1 TO numSections TREEVIEW GET CHECK tabSearchSettings, %IDC_SECTIONS_TREE, SectionArrayHandle(ctr) TO thisCheck IF thisCheck <> %FALSE THEN IF tempSectionList$ <> "" THEN tempSectionList$ = tempSectionList$ & "," tempSectionList$ = tempSectionList$ & SectionList$(ctr, 2) END IF NEXT 'Grab search terms CONTROL GET TEXT tabSearchSettings, %IDC_SEARCHTERMS_BOX TO tempSearchTerms$ 'And output it! IF INSTR(CurrentSearchname$, "\") = 0 THEN SearchFile$ = realEXEpath$() & CurrentSearchname$ & ".zen" 'Add local path if we don't have it already ELSE SearchFile$ = CurrentSearchname$ & ".zen" END IF OPEN SearchFile$ FOR OUTPUT AS #ff PRINT #ff, CurrentSearchname$ PRINT #ff, tempCityList$ PRINT #ff, tempSectionList$ PRINT #ff, tempSearchTerms$ 'Finally, output the results list FOR ctr = 1 TO numResults IF TRIM$(ResultsArray$(ctr, 2)) = "" THEN ITERATE FOR 'Bad item somehow, skip it PRINT #ff, """" & ResultsArray$(ctr, 1) & ""","; 'Date PRINT #ff, """" & CSVsafe$(ResultsArray$(ctr, 2)) & ""","; 'Title PRINT #ff, """" & ResultsArray$(ctr, 3) & ""","; 'City PRINT #ff, """" & ResultsArray$(ctr, 4) & ""","; 'Section PRINT #ff, """" & ResultsArray$(ctr, 5) & ""","; 'URI PRINT #ff, """" & CSVsafe$(ResultsArray$(ctr, 6)) & """" 'Full Text NEXT CLOSE #ff UnsavedResults = %FALSE END SUB SUB LoadSettings() LOCAL IniFile$ LOCAL ff AS LONG, ctr AS LONG LOCAL temp$, lineCmd$, lineData$, CityFile$, SectFile$ 'Import city list DIM CityList$(1 TO 1000, 1 TO 6) '1=region, 2=country code, 3=country name, 4=city name, 5=cl url, 6=status (-1/0) CityFile$ = realEXEpath$() & "CLcitylist.csv" IF ISFILE(CityFile$) <> %FALSE THEN ff = FREEFILE NumCities = 0 OPEN CityFile$ FOR INPUT AS #ff LINE INPUT #ff, temp$ DO UNTIL EOF(ff) LINE INPUT #ff, temp$ INCR NumCities FOR ctr = 1 TO 5 CityList$(NumCities, ctr) = PARSE$(temp$, ctr) NEXT LOOP CLOSE #ff END IF 'Import sections DIM SectionList$(1 TO 500, 1 TO 4) '1=supersection, 2=section, 3=name, 4=status (-1/0) SectFile$ = realEXEpath$() & "CLsectlist.csv" IF ISFILE(SectFile$) <> %FALSE THEN ff = FREEFILE NumSections = 0 OPEN SectFile$ FOR INPUT AS #ff LINE INPUT #ff, temp$ DO UNTIL EOF(ff) LINE INPUT #ff, temp$ INCR NumSections FOR ctr = 1 TO 3 SectionList$(NumSections, ctr) = PARSE$(temp$, ctr) NEXT LOOP CLOSE #ff END IF 'Load settings IniFile$ = realEXEpath$() & "zenclsh.ini" 'Set up defaults CurrentSearchname$ = "default" SearchCraigslist = %TRUE SearchKijiji = %FALSE AutoDeepScan = %FALSE CheckForUpdates = %TRUE DiscardSearches = %TRUE MaxResults = 25000 MaxThreads = 4 MaxThreads2 = 4 newX = 398 newY = 286 UserAgent$ = "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)" 'What browser are we faking? startMenu$ = "later" searchTitles = %FALSE hasPictures = %FALSE 'Try and load settings IF ISFILE(IniFile$) <> %FALSE THEN ff = FREEFILE OPEN IniFile$ FOR INPUT AS #ff DO UNTIL EOF(ff) LINE INPUT #ff, temp$ lineCmd$ = LCASE$(TRIM$(PARSE$(temp$, ":", 1))) lineData$ = TRIM$(PARSE$(temp$, ":", 2)) IF lineData$ <> "" THEN SELECT CASE lineCmd$ CASE "searchtitles" searchTitles = VAL(lineData$) CASE "haspictures" searchTitles = VAL(lineData$) CASE "width" newX = VAL(lineData$) CASE "height" newY = VAL(lineData$) CASE "currentsearchname" CurrentSearchName$ = lineData$ IF forceSearchName$ <> "" THEN CurrentSearchName$ = forceSearchName$ 'If a search name is being forced by the command line, force it... forceSearchName$ = "" ' ...but only force it once. END IF CASE "searchcraigslist" SearchCraigslist = VAL(lineData$) CASE "searchkijiji" SearchKijiji = VAL(lineData$) CASE "autodeepscan" AutoDeepScan = VAL(lineData$) CASE "checkforupdates" CheckForUpdates = VAL(lineData$) CASE "discardsearches" DiscardSearches = VAL(lineData$) CASE "startmenu" StartMenu$ = LCASE$(lineData$) CASE "maxresults" MaxResults = VAL(lineData$) IF MaxResults < 5000 THEN MaxResults = 5000 ELSEIF MaxResults > 1000000 THEN MaxResults = 1000000 END IF CASE "maxthreads" MaxThreads = VAL(lineData$) IF MaxThreads < 1 THEN MaxThreads = 1 ELSEIF MaxThreads > 12 THEN MaxThreads = 12 END IF CASE "maxthreads2" MaxThreads2 = VAL(lineData$) IF MaxThreads2 < 1 THEN MaxThreads2 = 1 ELSEIF MaxThreads2 > 12 THEN MaxThreads2 = 1 END IF CASE "useragent" UserAgent$ = TRIM$(lineData$) END SELECT END IF LOOP CLOSE #ff END IF END SUB 'Process clicks from buttons and other stuff CALLBACK FUNCTION ShowZENCLSHDIALOGProc() LOCAL ctr AS LONG, ctr2 AS LONG, res AS LONG, dummy AS LONG LOCAL iFailed AS LONG, ff AS LONG LOCAL itemFlipped AS DWORD, tempParent AS DWORD, foundSome AS LONG LOCAL ThisValue AS LONG LOCAL thisCheck AS LONG LOCAL tempFilename$, sFileSpec$ LOCAL countNew AS LONG, countOld AS LONG, outputMode AS LONG, recordCtr AS LONG LOCAL thisRow AS LONG, foundRow AS LONG, columnNo AS LONG LOCAL pNMLV AS NM_LISTVIEW POINTER LOCAL temp$ LOCAL pszURL AS ASCIIZ PTR LOCAL desktopHandle AS DWORD LOCAL QStack() AS LONG LOCAL LastElement AS LONG, FirstElement AS LONG, NumberOfElements AS LONG, StackPtr AS LONG LOCAL I AS LONG, J AS LONG SELECT CASE AS LONG CBMSG CASE %WM_INITDIALOG 'Init routines would go here if there were any... CASE %WM_SIZE DIALOG GET SIZE mainDialogHandle TO newX, newY 'Make sure dialog is of the minimize size in each dimension IF newX < 398 OR newY < 313 THEN 'Too small IF newX < 398 THEN newX = 398 'The odd thing is... for some reason the init size is 388 x 286 IF newY < 313 THEN newY = 313 ' ...perhaps it adds the caption/title and border to the size. Odd. Maybe. DIALOG SET SIZE mainDialogHandle, newX, newY END IF 'Now resize and move the controls as needed 'Move progress bar to bottom, change width CONTROL SET LOC mainDialogHandle, %IDC_PROGRESS, 5, newY - 41 CONTROL SET SIZE mainDialogHandle, %IDC_PROGRESS, newX - 23, 10 'Resize tab control CONTROL SET SIZE mainDialogHandle, %IDC_SYSTAB, newX - 23, newY - 80 'Search results tab: Move down Clear, Output CSV, View HTML, Deep Scan, and Go buttons CONTROL SET LOC tabSearchResults, %IDC_CLEARRESULTS, 2, newY - 115 CONTROL SET LOC tabSearchResults, %IDC_OUTPUT_CSV, 67, newY - 115 CONTROL SET LOC tabSearchResults, %IDC_VIEW_HTML, 122, newY - 115 CONTROL SET LOC tabSearchResults, %IDC_DEEPSCAN, newX - 157, newY - 115 CONTROL SET LOC tabSearchResults, %IDC_GO_SEARCH, newX - 82, newY - 115 'Search results tab: Resize listview CONTROL SET SIZE tabSearchResults, %IDC_RESULTSLIST, newX - 33, newY - 125 LISTVIEW SET COLUMN tabSearchResults, %IDC_RESULTSLIST, 2, newX - 253 CONTROL REDRAW tabSearchResults, %IDC_RESULTSLIST 'For some reason a listview must be redrawn after resize 'Search settings tab: Move down Search Terms label and textbox, and widen textbox CONTROL SET LOC tabSearchSettings, %IDC_LABEL, 2, newY - 113 CONTROL SET LOC tabSearchSettings, %IDC_SEARCHTERMS_BOX, 55, newY - 115 CONTROL SET SIZE tabSearchSettings, %IDC_SEARCHTERMS_BOX, NewX - 85, 15 'Search settings tab: Move down Select All, Clear All, Select All, and Clear All buttons CONTROL SET LOC tabSearchSettings, %IDC_CITIESALL, 2, newY - 135 CONTROL SET LOC tabSearchSettings, %IDC_CITIES_CLEARALL, 57, newY - 135 CONTROL SET LOC tabSearchSettings, %IDC_SECTIONSELECTALL, ((newX - 20) \ 2), newY - 135 CONTROL SET LOC tabSearchSettings, %IDC_SECTION_CLEARALL, ((newX - 20) \ 2) + 55, newY - 135 'Search settings tab: Resize cities treeview, Move and resize sections treeview CONTROL SET LOC tabSearchSettings, %IDC_SECTIONS_TREE, ((newX - 20) \ 2), 5 CONTROL SET SIZE tabSearchSettings, %IDC_SECTIONS_TREE, ((newX - 20) \ 2) - 10, newY - 145 CONTROL SET SIZE tabSearchSettings, %IDC_CITIESTREE, ((newX - 20) \ 2) - 10, newY - 145 'Software settings tab: Move down Cancel, Apply, and Visit Website buttons CONTROL SET LOC tabSoftwareSettings, %IDC_APPLY_SETTINGS, 57, newY - 115 CONTROL SET LOC tabSoftwareSettings, %IDC_SETTINGS_CANCEL, 2, newY - 115 CONTROL SET LOC tabSoftwareSettings, %IDC_VISITWEB, newX - 92, newY - 115 'Diagnostics tab needs no changes, but we might as well widen the labels anyway FOR ctr = 1 TO 12 CONTROL SET SIZE tabDiagnostics, %IDC_STHREAD + ctr, newX - 60, 10 NEXT FOR ctr = 1 TO 12 CONTROL SET SIZE tabDiagnostics, %IDC_DTHREAD + ctr, newX - 60, 10 NEXT CASE %WM_SYSCOMMAND IF (CBWPARAM AND &HFFF0) = %SC_CLOSE THEN 'Trying to close window IF UnsavedResults = 1 THEN 'Unsaved results (which could include a search that's running) res = MSGBOX("You have unsaved results. Are you sure you want to exit without saving?", %MB_YESNO + %MB_APPLMODAL + %MB_ICONQUESTION, "Really exit?") IF res <> %IDYES THEN FUNCTION = 1 'Abort close EXIT FUNCTION END IF END IF SearchAbort = %TRUE 'Just in case DeepScanAbort = %TRUE 'Not sure if threads auto-die, but this'll kill them WriteSettings() IF aboutDialogHandle <> 0 THEN DIALOG END aboutDialogHandle 'Close about window if it's still open END IF DIALOG END mainDialogHandle ELSE FUNCTION = 0 END IF CASE %WM_NCACTIVATE STATIC hWndSaveFocus AS DWORD IF ISFALSE CBWPARAM THEN ' Save control focus hWndSaveFocus = GetFocus() ELSEIF hWndSaveFocus THEN ' Restore control focus SetFocus(hWndSaveFocus) hWndSaveFocus = 0 END IF CASE %WM_COMMAND ' Process control notifications SELECT CASE AS LONG CBCTL CASE %IDC_LOADSEARCH 'Load search button IF CBCTLMSG = %BN_CLICKED OR CBCTLMSG = 1 THEN iFailed = %FALSE 'Is the scan currently running? (Must be stopped first) IF SearchRunning <> %FALSE OR DeepScanRunning <> %FALSE THEN MSGBOX "A search is currently active. You need to stop the current scan before loading an old one.", %MB_SYSTEMMODAL OR %MB_ICONERROR, "Error" TAB SELECT mainDialogHandle, %IDC_SYSTAB, 1 iFailed = %TRUE END IF 'Are there unsaved results? (Confirm before we lose them) IF UnsavedResults <> %FALSE AND iFailed = %FALSE THEN thisCheck = MSGBOX("You have unsaved search results. Would you like to load anyway and lose these results?", %MB_SYSTEMMODAL OR %MB_YESNO, "Warning") IF thisCheck <> %IDYES THEN iFailed = %TRUE END IF END IF 'Load it IF iFailed = %FALSE THEN CONTROL GET TEXT mainDialogHandle, %IDC_SEARCHNAME_BOX TO sFileSpec$ sFileSpec$ = sFileSpec$ & ".zen" DISPLAY OPENFILE mainDialogHandle, , , "Please select a search file", realEXEpath$(), _ "All ZenCLSH Searches (*.zen)" & CHR$(0) & "*.zen" & CHR$(0), sFileSpec$, "zen", _ %OFN_FILEMUSTEXIST OR %OFN_PATHMUSTEXIST OR %OFN_ENABLESIZING OR %OFN_FILEMUSTEXIST TO TempFilename$ TempFilename$ = TRIM$(TempFilename$) REPLACE ".zen" WITH "" IN TempFilename$ TempFilename$ = MID$(TempFilename$, INSTR(-1, TempFilename$, "\") + 1) IF LEN(TempFilename$) <> 0 THEN CONTROL SET TEXT mainDialogHandle, %IDC_SEARCHNAME_BOX, TempFilename$ LoadSearch %FALSE END IF END IF END IF CASE %IDC_SAVESEARCH 'Save search button IF CBCTLMSG = %BN_CLICKED OR CBCTLMSG = 1 THEN 'Save it CONTROL GET TEXT mainDialogHandle, %IDC_SEARCHNAME_BOX TO sFileSpec$ sFileSpec$ = sFileSpec$ & ".zen" DISPLAY SAVEFILE mainDialogHandle, , , "Please select a search file", realEXEpath$(), _ "All ZenCLSH Searches (*.zen)" & CHR$(0) & "*.zen" & CHR$(0), sFileSpec$, "zen", _ %OFN_FILEMUSTEXIST OR %OFN_PATHMUSTEXIST OR %OFN_ENABLESIZING OR %OFN_OVERWRITEPROMPT OR %OFN_CREATEPROMPT TO TempFilename$ TempFilename$ = TRIM$(TempFilename$) REPLACE ".zen" WITH "" IN TempFilename$ TempFilename$ = MID$(TempFilename$, INSTR(-1, TempFilename$, "\") + 1) IF LEN(TempFilename$) <> 0 THEN CONTROL SET TEXT mainDialogHandle, %IDC_SEARCHNAME_BOX, TempFilename$ UnsavedResults = %FALSE SaveSearch() WriteSettings() ELSE MSGBOX "Save cancelled. Searches have NOT been saved.", %MB_SYSTEMMODAL OR %MB_ICONERROR, "Aborted" END IF END IF CASE %IDC_CLEARRESULTS IF CBCTLMSG = %BN_CLICKED OR CBCTLMSG = 1 THEN 'Is a scan of some type running? (Must be stopped first) IF SearchRunning <> %FALSE AND DeepScanRunning <> %FALSE THEN MSGBOX "If you want to clear the results, please stop the search first.", %MB_SYSTEMMODAL OR %MB_ICONERROR, "Error" iFailed = %TRUE END IF 'Is there even anything to clear? LISTVIEW GET COUNT tabSearchResults, %IDC_RESULTSLIST TO thisCheck 'How many items are there? IF thisCheck = 0 THEN iFailed = %TRUE 'Skip clearing, no need 'Confirm results clearing IF iFailed <> %TRUE THEN thisCheck = MSGBOX("Are you sure you want to premanently delete your search results?", %MB_SYSTEMMODAL OR %MB_YESNO, "Warning") IF thisCheck <> %IDNO THEN 'Clear listview box and clear array that holds the results 'Erase ResultsArray$() and ptr ERASE ResultsArray$() REDIM ResultsArray$(1 TO MaxResults, 1 TO 7) numResults = 0 'Erase results listview LISTVIEW RESET tabSearchResults, %IDC_RESULTSLIST END IF END IF END IF CASE %IDC_OUTPUT_CSV IF CBCTLMSG = %BN_CLICKED OR CBCTLMSG = 1 THEN 'See if there are multiple selections in the listbox LISTVIEW GET SELCOUNT tabSearchResults, %IDC_RESULTSLIST TO countNew 'See how many items are selected IF countNew > 1 THEN 'If it's more than one... outputMode = 3 ' ...go into "selected items" mode RESET SelectedItems() 'Clear the "item is selected" array thisRow = 1 'Start searching at the beginning DO LISTVIEW GET SELECT tabSearchResults, %IDC_RESULTSLIST, thisRow TO foundRow 'Find the next selected row IF foundRow = 0 THEN EXIT DO 'No more selected items SelectedItems(foundRow) = %TRUE 'Set it to selected IF foundRow = numResults THEN EXIT DO 'Have we reached the end of the array? Exit if so thisRow = foundRow + 1 'Continue search at the next item LOOP ELSE countNew = 0 'Reset old/new counters countOld = 0 FOR ctr = 1 TO numResults 'Go through all the results and see what's old and what's new IF ResultsArray$(ctr, 7) = "" THEN 'This is an old entry INCR countOld ELSE '...and this one was found in today's session INCR countNew END IF NEXT IF countOld + countNew = 0 THEN 'Hey... there's nothing to output anyway! MSGBOX "There are no records to save. I can not count your chickens before they've hatched.", %MB_SYSTEMMODAL OR %MB_ICONERROR, "Aborted" outputMode = 0 'No output EXIT SELECT END IF IF countNew > 0 AND countOld > 0 THEN 'We have new record, and we have old records temp$ = "Would you like to output only the " & FORMAT$(countNew, "0,") & " new record" IF countNew > 1 THEN temp$ = temp$ & "s" ' spell it right: "1 record" or "2+ records" temp$ = temp$ & "? (If not, everything will be included in the CSV file)." res = MSGBOX(temp$, %MB_YESNOCANCEL, "Select Scope") 'Ask them IF res = %IDYES THEN outputMode = 2 'Output new only ELSEIF res = %IDNO THEN outputMode = 1 'Output everything ELSE 'must be %IDCANCEL MSGBOX "Save cancelled. CSV file has NOT been saved.", %MB_SYSTEMMODAL OR %MB_ICONERROR, "Aborted" outputMode = 0 EXIT SELECT END IF ELSE 'The records are either all old, or all new, so we output everything outputMode = 1 'Output everything END IF END IF recordCtr = 0 'Spit out CSV file with all information CONTROL GET TEXT mainDialogHandle, %IDC_SEARCHNAME_BOX TO sFileSpec$ sFileSpec$ = sFileSpec$ & ".csv" DISPLAY SAVEFILE mainDialogHandle, , , "Please select a CSV file", realEXEpath$(), _ "All CSV files (*.csv)" & CHR$(0) & "*.csv" & CHR$(0), sFileSpec$, "csv", _ %OFN_FILEMUSTEXIST OR %OFN_PATHMUSTEXIST OR %OFN_ENABLESIZING OR %OFN_OVERWRITEPROMPT OR %OFN_CREATEPROMPT TO TempFilename$ TempFilename$ = TRIM$(TempFilename$) 'Grab CSV filename; no cropping needed here IF LEN(TempFilename$) <> 0 THEN ff = FREEFILE OPEN TempFilename$ FOR OUTPUT AS #ff 'Open output line PRINT #ff, """Date"",""Title"",""City"",""Section"",""URI"",""FullText""" 'Write header line FOR ctr = 1 TO numResults 'Go through all results IF outputMode = 2 AND ResultsArray$(ctr, 7) = "" THEN ITERATE FOR 'If we're new only, skip old records IF outputMode = 3 AND SelectedItems(ctr) = %FALSE THEN ITERATE FOR 'If we're selected only, skip ones that aren't selected INCR recordCtr FOR ctr2 = 1 TO 6 'Go through all elements PRINT #ff, """" & CSVsafe$((ResultsArray$(ctr, ctr2))) & """"; 'Write to file IF ctr2 <> numResults THEN PRINT #ff, ","; 'More terms left ELSE PRINT #ff, "" 'Line end END IF NEXT temp$ = "CSV file saved to '" & TempFilename$ & "' with " & FORMAT$(recordCtr, "0,") & " record" 'create success message IF recordCtr > 1 THEN temp$ = temp$ & "s" 'I hate it when an app says "success: 1 records saved"... lazy temp$ = temp$ & "." MSGBOX temp$, %MB_OK, "Save complete" NEXT CLOSE #ff ELSE MSGBOX "Save cancelled. CSV file has NOT been saved.", %MB_SYSTEMMODAL OR %MB_ICONERROR, "Aborted" END IF END IF CASE %IDC_VIEW_HTML IF CBCTLMSG = %BN_CLICKED OR CBCTLMSG = 1 THEN LISTVIEW GET SELCOUNT tabSearchResults, %IDC_RESULTSLIST TO countNew 'See how many items are selected IF countNew > 1 THEN 'If it's more than one... outputMode = 3 ' ...go into "selected items" mode RESET SelectedItems() 'Clear the "item is selected" array thisRow = 1 'Start searching at the beginning DO LISTVIEW GET SELECT tabSearchResults, %IDC_RESULTSLIST, thisRow TO foundRow 'Find the next selected row IF foundRow = 0 THEN EXIT DO 'No more selected items SelectedItems(foundRow) = %TRUE 'Set it to selected IF foundRow = numResults THEN EXIT DO 'Have we reached the end of the array? Exit if so thisRow = foundRow + 1 'Continue search at the next item LOOP ELSE countNew = 0 'Reset old/new counters countOld = 0 FOR ctr = 1 TO numResults 'Go through all the results and see what's old and what's new IF ResultsArray$(ctr, 7) = "" THEN 'This is an old entry INCR countOld ELSE '...and this one was found in today's session INCR countNew END IF NEXT IF countOld + countNew = 0 THEN 'Hey... there's nothing to output anyway! MSGBOX "There are no records to view. I can not count your chickens before they've hatched.", %MB_SYSTEMMODAL OR %MB_ICONERROR, "Aborted" outputMode = 0 'No output EXIT SELECT ELSEIF countNew > 0 AND countOld > 0 THEN 'We have new record, and we have old records temp$ = "Would you like to view only the " & FORMAT$(countNew, "0,") & " new record" IF countNew > 1 THEN temp$ = temp$ & "s" ' spell it right: "1 record" or "2+ records" temp$ = temp$ & "? (If not, everything will be included in the file)." res = MSGBOX(temp$, %MB_YESNOCANCEL, "Select Scope") 'Ask them IF res = %IDYES THEN outputMode = 2 'Output new only ELSEIF res = %IDNO THEN outputMode = 1 'Output everything ELSE 'must be %IDCANCEL outputMode = 0 EXIT SELECT END IF ELSE 'The records are either all old, or all new, so we output everything outputMode = 1 'Output everything END IF END IF recordCtr = 0 'Spit out HTML file with all information needed CONTROL GET TEXT mainDialogHandle, %IDC_SEARCHNAME_BOX TO sFileSpec$ sFileSpec$ = sFileSpec$ & ".html" DISPLAY SAVEFILE mainDialogHandle, , , "Please select an HTML file", realEXEpath$(), _ "All HTML files (*.html)" & CHR$(0) & "*.html" & CHR$(0), sFileSpec$, "html", _ %OFN_FILEMUSTEXIST OR %OFN_PATHMUSTEXIST OR %OFN_ENABLESIZING OR %OFN_OVERWRITEPROMPT OR %OFN_CREATEPROMPT TO TempFilename$ TempFilename$ = TRIM$(TempFilename$) 'Grab HTML filename; no cropping needed here IF LEN(TempFilename$) <> 0 THEN OutputHTML TempFilename$, outputMode 'Output an HTML file with all the results in it pszURL = STRPTR(TempFilename$) desktopHandle = GetDesktopWindow() res = ShellExecute(desktopHandle, "open", @pszURL, "", "", %SW_SHOWDEFAULT) END IF END IF CASE %IDC_DEEPSCAN IF CBCTLMSG = %BN_CLICKED OR CBCTLMSG = 1 THEN IF DeepScanRunning <> %FALSE THEN 'Scan is currently running; DeepScanAbort = %TRUE ' ...abort ELSE 'Not running; start new scan DeepScanRunning = %TRUE 'Generate a list of all pages, from most recent to oldest, that need to be downloaded foundSome = 0 FOR ctr = 1 TO numResults IF LEN(ResultsArray$(ctr, 6)) < 50 THEN 'Has this one been done yet? INCR foundSome END IF NEXT IF foundSome = 0 THEN 'Might have nothing to do MSGBOX "Everything has already been downloaded, so there's no need to do a deep scan.", ,"No worries" ELSE THREAD CREATE doLaunchDeep(dummy) TO lThread??? END IF END IF END IF CASE %IDC_SEARCHTERMS_BOX 'Search term box has been typed into... 'if CBCTLMSG = %EN_CHANGE or cbctlmsg = %EN_UPDATE then ' CONTROL GET TEXT tabSearchSettings, %IDC_SEARCHTERMS_BOX TO thisTermList$ 'end if CASE %IDC_GO_SEARCH 'Start search, or abort it, depending on button state IF CBCTLMSG = %BN_CLICKED OR CBCTLMSG = 1 THEN IF SearchRunning = %FALSE THEN iFailed = %FALSE 'Make sure that everything is set 'Does the search have a name? CONTROL GET TEXT mainDialogHandle, %IDC_SEARCHNAME_BOX TO thisSearchName$ thisSearchName$ = TRIM$(thisSearchName$) IF thisSearchName$ = "" THEN MSGBOX "You have not set a search title. Please do that first.", %MB_SYSTEMMODAL OR %MB_ICONERROR, "Error" iFailed = %TRUE END IF 'Are terms set? IF iFailed = %FALSE THEN CONTROL GET TEXT tabSearchSettings, %IDC_SEARCHTERMS_BOX TO doTermList$ doTermList$ = TRIM$(doTermList$) IF doTermList$ = "" THEN MSGBOX "You have not set any search terms. Please do that first.", %MB_SYSTEMMODAL OR %MB_ICONERROR, "Error" TAB SELECT mainDialogHandle, %IDC_SYSTAB, 2 iFailed = %TRUE END IF END IF 'Are regions set? IF iFailed = %FALSE THEN doCityList$ = "" FOR ctr = 1 TO numCities 'Go through all the cities IF CityArrayStatus(ctr) <> %FALSE THEN CityList$(ctr, 6) = "1" IF doCityList$ <> "" THEN doCityList$ = doCityList$ & "," doCityList$ = doCityList$ & CityList$(ctr, 5) 'Add search URL to our to-do list ELSE CityList$(ctr, 6) = "0" END IF NEXT IF doCityList$ = "" THEN MSGBOX "You have not selected any cities or regions to search. Please do that first.", %MB_SYSTEMMODAL OR %MB_ICONERROR, "Error" TAB SELECT mainDialogHandle, %IDC_SYSTAB, 2 iFailed = %TRUE END IF END IF 'Are sections set? IF iFailed = %FALSE THEN doSectionList$ = "" FOR ctr = 1 TO numSections TREEVIEW GET CHECK tabSearchSettings, %IDC_SECTIONS_TREE, SectionArrayHandle(ctr) TO thisCheck IF thisCheck <> %FALSE THEN SectionList$(ctr, 4) = FORMAT$(ABS(%TRUE)) IF doSectionList$ <> "" THEN doSectionList$ = doSectionList$ & "," doSectionList$ = doSectionList$ & SectionList$(ctr, 2) ELSE SectionList$(ctr, 4) = FORMAT$(%FALSE) END IF NEXT IF doSectionList$ = "" THEN MSGBOX "You have not selected any sections to search. Please do that first.", %MB_SYSTEMMODAL OR %MB_ICONERROR, "Error" TAB SELECT mainDialogHandle, %IDC_SYSTAB, 2 iFailed = %TRUE END IF END IF 'Is search system running? IF iFailed = %FALSE THEN IF SearchRunning = %TRUE THEN MSGBOX "A search is currently running. Please abort it and wait a moment for it to stop before starting a new one.", %MB_SYSTEMMODAL OR %MB_ICONERROR, "Error" iFailed = %TRUE END IF END IF IF iFailed = %FALSE THEN 'Turn on search system (turned off by function completing/aborting) SearchRunning = %TRUE SearchAbort = %FALSE 'Switch search button to "Abort" CONTROL SET TEXT tabSearchResults, %IDC_GO_SEARCH, "Abort" 'Disable relevant parts of form (cities, terms, sections) CONTROL DISABLE mainDialogHandle, %IDC_LOADSEARCH CONTROL DISABLE mainDialogHandle, %IDC_SEARCHNAME_BOX CONTROL DISABLE tabSearchSettings, %IDC_CITIESTREE CONTROL DISABLE tabSearchSettings, %IDC_CITIESALL CONTROL DISABLE tabSearchSettings, %IDC_CITIES_CLEARALL CONTROL DISABLE tabSearchSettings, %IDC_SECTIONS_TREE CONTROL DISABLE tabSearchSettings, %IDC_SECTIONSELECTALL CONTROL DISABLE tabSearchSettings, %IDC_SECTION_CLEARALL CONTROL DISABLE tabSearchSettings, %IDC_SEARCHTERMS_BOX CONTROL DISABLE tabSoftwareSettings, %IDC_CHECK_CL CONTROL DISABLE tabSoftwareSettings, %IDC_CHECK_KIJIJI CONTROL DISABLE tabSoftwareSettings, %IDC_CHECK_AUTODEEP CONTROL DISABLE tabSoftwareSettings, %IDC_SETTINGS_CANCEL CONTROL DISABLE tabSoftwareSettings, %IDC_APPLY_SETTINGS 'Launch search thread THREAD CREATE DoHarvest(dummy) TO hThread??? END IF ELSE 'ABORT SEARCH... Works by sending a signal to the search function to stop ASAP 'Flip abort switch (button will be switched back to "go" by other tool, hopefully) SearchAbort = %TRUE IF SearchRunning = %FALSE THEN 'something has malfunctioned (search probably terminated normally), manually set it back on 'Switch controls back to what they should be CONTROL SET TEXT tabSearchResults, %IDC_GO_SEARCH, "Go!" CONTROL ENABLE mainDialogHandle, %IDC_LOADSEARCH CONTROL ENABLE mainDialogHandle, %IDC_SEARCHNAME_BOX CONTROL ENABLE tabSearchSettings, %IDC_CITIESTREE CONTROL ENABLE tabSearchSettings, %IDC_CITIESALL CONTROL ENABLE tabSearchSettings, %IDC_CITIES_CLEARALL CONTROL ENABLE tabSearchSettings, %IDC_SECTIONS_TREE CONTROL ENABLE tabSearchSettings, %IDC_SECTIONSELECTALL CONTROL ENABLE tabSearchSettings, %IDC_SECTION_CLEARALL CONTROL ENABLE tabSearchSettings, %IDC_SEARCHTERMS_BOX CONTROL ENABLE tabSoftwareSettings, %IDC_CHECK_CL 'CONTROL enable tabSoftwareSettings, %IDC_CHECK_KIJIJI 'Not currently active IF DeepScanRunning = %FALSE THEN CONTROL ENABLE tabSoftwareSettings, %IDC_CHECK_AUTODEEP END IF CONTROL ENABLE tabSoftwareSettings, %IDC_SETTINGS_CANCEL CONTROL ENABLE tabSoftwareSettings, %IDC_APPLY_SETTINGS END IF 'Deactivate thread THREAD CLOSE hThread??? TO dummy END IF END IF CASE %IDC_CITIESALL 'Select all cities IF CBCTLMSG = %BN_CLICKED OR CBCTLMSG = 1 THEN CONTROL SEND CB.HNDL, %IDC_CITIESTREE, %WM_SETREDRAW, %FALSE, 0 'Flip on all regions FOR ctr = 1 TO tempNumRegions IF RegionArrayStatus(ctr) = %FALSE THEN TREEVIEW SET CHECK CB.HNDL, %IDC_CITIESTREE, RegionArrayHandle(ctr), %TRUE RegionArrayStatus(ctr) = ABS(%TRUE) END IF NEXT 'Flip on all countries FOR ctr = 1 TO tempNumCountries IF CountryArrayStatus(ctr) = %FALSE THEN TREEVIEW SET CHECK CB.HNDL, %IDC_CITIESTREE, CountryArrayHandle(ctr), %TRUE CountryArrayStatus(ctr) = ABS(%TRUE) END IF NEXT 'Flip on all cities FOR ctr = 1 TO numCities IF CityArrayStatus(ctr) = %FALSE THEN TREEVIEW SET CHECK CB.HNDL, %IDC_CITIESTREE, CityArrayHandle(ctr), %TRUE CityArrayStatus(ctr) = ABS(%TRUE) END IF NEXT CONTROL SEND CB.HNDL, %IDC_CITIESTREE, %WM_SETREDRAW, %TRUE, 0 END IF CASE %IDC_CITIES_CLEARALL 'Unselect all cities IF CBCTLMSG = %BN_CLICKED OR CBCTLMSG = 1 THEN CONTROL SEND CB.HNDL, %IDC_CITIESTREE, %WM_SETREDRAW, %FALSE, 0 'Flip off all regions FOR ctr = 1 TO tempNumRegions IF RegionArrayStatus(ctr) <> %FALSE THEN TREEVIEW SET CHECK CB.HNDL, %IDC_CITIESTREE, RegionArrayHandle(ctr), %FALSE RegionArrayStatus(ctr) = %FALSE END IF NEXT 'Flip off all countries FOR ctr = 1 TO tempNumCountries IF CountryArrayStatus(ctr) <> %FALSE THEN TREEVIEW SET CHECK CB.HNDL, %IDC_CITIESTREE, CountryArrayHandle(ctr), %FALSE CountryArrayStatus(ctr) = %FALSE END IF NEXT 'Flip off all cities FOR ctr = 1 TO numCities IF CityArrayStatus(ctr) <> %FALSE THEN TREEVIEW SET CHECK CB.HNDL, %IDC_CITIESTREE, CityArrayHandle(ctr), %FALSE CityArrayStatus(ctr) = %FALSE END IF NEXT CONTROL SEND CB.HNDL, %IDC_CITIESTREE, %WM_SETREDRAW, %TRUE, 0 END IF CASE %IDC_SECTIONSELECTALL 'Select all sections IF CBCTLMSG = %BN_CLICKED OR CBCTLMSG = 1 THEN CONTROL SEND CB.HNDL, %IDC_SECTIONS_TREE, %WM_SETREDRAW, %FALSE, 0 FOR ctr = 1 TO numSections 'Select all sections TREEVIEW GET PARENT CB.HNDL, %IDC_SECTIONS_TREE, SectionArrayHandle(ctr) TO tempParent IF tempParent = 0 THEN 'It's top-level (select it) TREEVIEW SET CHECK CB.HNDL, %IDC_SECTIONS_TREE, SectionArrayHandle(ctr), %TRUE SectionList$(ctr, 4) = "-1" ELSE 'It's a child, it can be ignored since the parent covers it TREEVIEW SET CHECK CB.HNDL, %IDC_SECTIONS_TREE, SectionArrayHandle(ctr), %FALSE SectionList$(ctr, 4) = "0" END IF NEXT CONTROL SEND CB.HNDL, %IDC_SECTIONS_TREE, %WM_SETREDRAW, %TRUE, 0 END IF CASE %IDC_SECTION_CLEARALL 'Deselect all sections IF CBCTLMSG = %BN_CLICKED OR CBCTLMSG = 1 THEN CONTROL SEND CB.HNDL, %IDC_SECTIONS_TREE, %WM_SETREDRAW, %FALSE, 0 FOR ctr = 1 TO numSections 'Deselect all sections TREEVIEW SET CHECK CB.HNDL, %IDC_SECTIONS_TREE, SectionArrayHandle(ctr), %FALSE SectionList$(ctr, 4) = "0" NEXT CONTROL SEND CB.HNDL, %IDC_SECTIONS_TREE, %WM_SETREDRAW, %TRUE, 0 END IF CASE %IDC_APPLY_SETTINGS 'Apply and save settings to disk IF CBCTLMSG = %BN_CLICKED OR CBCTLMSG = 1 THEN 'Grab settings CONTROL GET CHECK tabSoftwareSettings, %IDC_CHECK_CL TO SearchCraigslist CONTROL GET CHECK tabSoftwareSettings, %IDC_CHECK_KIJIJI TO SearchKijiji CONTROL GET CHECK tabSoftwareSettings, %IDC_CHECK_AUTODEEP TO AutoDeepScan CONTROL GET CHECK tabSoftwareSettings, %IDC_CHECKZEN TO CheckForUpdates CONTROL GET CHECK tabSoftwareSettings, %IDC_DISCARD TO DiscardSearches CONTROL GET CHECK tabSoftwareSettings, %IDC_CHECK_SEARCHTITLES TO searchTitles CONTROL GET CHECK tabSoftwareSettings, %IDC_CHECK_HASPICTURES TO hasPictures CONTROL GET TEXT mainDialogHandle, %IDC_SEARCHNAME_BOX TO CurrentSearchname$ CONTROL GET TEXT tabSoftwareSettings, %IDC_MAXRESULTS_BOX TO temp$ MaxResults = VAL(temp$) IF MaxResults < 5000 THEN MaxResults = 5000 CONTROL SET TEXT tabSoftwareSettings, %IDC_MAXRESULTS_BOX, FORMAT$(MaxResults, "0") ELSEIF MaxResults > 1000000 THEN MaxResults = 1000000 CONTROL SET TEXT tabSoftwareSettings, %IDC_MAXRESULTS_BOX, FORMAT$(MaxResults, "0") END IF CONTROL GET TEXT tabSoftwareSettings, %IDC_THREADSRESULTS_BOX TO temp$ MaxThreads = VAL(temp$) IF MaxThreads < 1 THEN MaxThreads = 1 CONTROL SET TEXT tabSoftwareSettings, %IDC_THREADSRESULTS_BOX, FORMAT$(MaxThreads, "0") ELSEIF MaxThreads > 12 THEN MaxThreads = 12 CONTROL SET TEXT tabSoftwareSettings, %IDC_THREADSRESULTS_BOX, FORMAT$(MaxThreads, "0") END IF CONTROL GET TEXT tabSoftwareSettings, %IDC_THREADS2RESULTS_BOX TO temp$ MaxThreads2 = VAL(temp$) IF MaxThreads2 < 1 THEN MaxThreads2 = 1 CONTROL SET TEXT tabSoftwareSettings, %IDC_THREADS2RESULTS_BOX, FORMAT$(MaxThreads2, "0") ELSEIF MaxThreads2 > 12 THEN MaxThreads2 = 12 CONTROL SET TEXT tabSoftwareSettings, %IDC_THREADS2RESULTS_BOX, FORMAT$(MaxThreads2, "0") END IF CONTROL GET TEXT tabSoftwareSettings, %IDC_USERAGENT_BOX TO UserAgent$ 'Save settings WriteSettings() END IF CASE %IDC_SETTINGS_CANCEL 'Cancel settings and restore to default saved values IF CBCTLMSG = %BN_CLICKED OR CBCTLMSG = 1 THEN 'Note: ideally this should happen when we leave the tab... To-do... 'Restore settings back to normal CONTROL SET CHECK tabSoftwareSettings, %IDC_CHECK_CL, SearchCraigslist CONTROL SET CHECK tabSoftwareSettings, %IDC_CHECK_KIJIJI, SearchKijiji CONTROL SET CHECK tabSoftwareSettings, %IDC_CHECK_AUTODEEP, AutoDeepScan CONTROL SET CHECK tabSoftwareSettings, %IDC_CHECKZEN, CheckForUpdates CONTROL SET CHECK tabSoftwareSettings, %IDC_DISCARD, DiscardSearches CONTROL SET CHECK tabSoftwareSettings, %IDC_CHECK_SEARCHTITLES, searchTitles CONTROL SET CHECK tabSoftwareSettings, %IDC_CHECK_HASPICTURES, hasPictures CONTROL SET TEXT tabSoftwareSettings, %IDC_MAXRESULTS_BOX, FORMAT$(MaxResults, "0") CONTROL SET TEXT tabSoftwareSettings, %IDC_THREADSRESULTS_BOX, FORMAT$(MaxThreads, "0") CONTROL SET TEXT tabSoftwareSettings, %IDC_THREADS2RESULTS_BOX, FORMAT$(MaxThreads2, "0") CONTROL SET TEXT tabSoftwareSettings, %IDC_USERAGENT_BOX, UserAgent$ END IF CASE %IDC_VISITWEB 'Visit Zentastic.com ZenCLSH homepage IF CBCTLMSG = %BN_CLICKED OR CBCTLMSG = 1 THEN ShowABOUTDIALOG %HWND_DESKTOP END IF END SELECT CASE %WM_HELP DIM lphi AS HELPINFO PTR lphi = CBLPARAM temp$ = "" SELECT CASE @lphi.iCtrlId 'Returns the ID of the item for which help is requested CASE %IDC_GO_SEARCH temp$ = "Click this button to begin a new search, or if a search is already running, " temp$ = temp$ & "click it to abort the current search." CASE %IDC_LOADSEARCH temp$ = "Click this to load the results and settings of an older search." CASE %IDC_SAVESEARCH temp$ = "Click this to save the results and settings of this search so you can work on it more later." CASE %IDC_PROGRESS temp$ = "This tells you the progress of the current search." CASE %IDC_CLEARRESULTS temp$ = "This deletes all current search results." CASE %IDC_OUTPUT_CSV temp$ = "This outputs a CSV (comma separated volume) file of the current search results (either all, new, or selected items) for easy " temp$ = temp$ & "loading into spreadsheets, databases, and other programs." CASE %IDC_VIEW_HTML temp$ = "This loads the results of your search into your browser (either all, new, or selected items)." CASE %IDC_DEEPSCAN temp$ = "This goes through all your search results and downloads the result pages so you can view " temp$ = temp$ & "or search them offline." CASE %IDC_RESULTSLIST temp$ = "This is the list of current results. You can double click an individual result to view it " temp$ = temp$ & "in your browser. You can also select multiple items to output or view specific records with the buttons below. " temp$ = temp$ & "Clicking on the column headers sorts the list." CASE %IDC_CITIESTREE temp$ = "This lets you select which cities or regions you'd like to search." CASE %IDC_CITIESALL temp$ = "Click this button to select all cities for searching." CASE %IDC_CITIES_CLEARALL temp$ = "Click this button to deselect all cities from searching." CASE %IDC_SECTIONS_TREE temp$ = "This lets you select which sections will be searched. Selecting a parent section automatically " temp$ = temp$ & " searches all sections within it." CASE %IDC_SECTIONSELECTALL temp$ = "Click this button to select all sections for searching." CASE %IDC_SECTION_CLEARALL temp$ = "Click this button to deselect all sections from searching." CASE %IDC_SEARCHTERMS_BOX temp$ = "Enter the terms you'd like to search for in this box, separating multiple terms with commas." CASE %IDC_SEARCHNAME_BOX temp$ = "Enter a name for your search in this box." CASE %IDC_CHECK_CL temp$ = "Check this box if you'd like to include Craigslist in your searching." CASE %IDC_CHECK_KIJIJI temp$ = "Check this box if you'd like to include Kijiji in your searching." CASE %IDC_CHECK_AUTODEEP temp$ = "Check this box to automatically download the entries for each item as well as just " temp$ = temp$ & "the search results." CASE %IDC_CHECKZEN temp$ = "Check this box if you'd like to have the software check for updates and tell you when " temp$ = temp$ & "they are available. Note that no updates will be automatically performed." CASE %IDC_DISCARD temp$ = "Check this box to automatically discard search results that are more than two months old." CASE %IDC_CHECK_SEARCHTITLES temp$ = "Check this box to only search the titles of ads (otherwise the text of the ad is searched as well)." CASE %IDC_CHECK_HASPICTURES temp$ = "Check this box to only search for ads that have pictures." CASE %IDC_MAXRESULTS_BOX temp$ = "Set the maximum number of results to allow (the more you allow, the more memory the software will allocate). Recommended default is 25000." CASE %IDC_THREADSRESULTS_BOX temp$ = "Set the maximum number of simultaneous internet connects to open per search. The more you use the faster it should go, although there is a maximum at which no gains are found." CASE %IDC_THREADS2RESULTS_BOX temp$ = "Set the maximum number of simultaneous internet connects to open per deep scan session. The more you use the faster it should go, although there is a maximum at which no gains are found." CASE %IDC_USERAGENT_BOX temp$ = "This sets the browser user agent string that it will pretend to be." CASE %IDC_SETTINGS_CANCEL temp$ = "Click this to abort any changes you've made to the settings." CASE %IDC_APPLY_SETTINGS temp$ = "Click this to apply/save the changes you've made to the settings (if you don't click this, " temp$ = temp$ & "the changes will be discarded)." CASE %IDC_VISITWEB temp$ = "Click this to visit the homepage of this software using your browser." END SELECT IF temp$ <> "" THEN ShowPopupHelp temp$ END IF CASE %WM_NOTIFY SELECT CASE CB.NMID CASE %IDC_RESULTSLIST IF CB.NMCODE = %NM_DBLCLK THEN 'Double click on results list LISTVIEW GET SELECT tabSearchResults, %IDC_RESULTSLIST TO thisCheck 'Get the current entry LISTVIEW GET USER tabSearchResults, %IDC_RESULTSLIST, thisCheck TO ThisValue 'Figure out which ResultsArray$() element it is 'Ask user if they'd like to view it res = MSGBOX("View this " & ResultsArray$(ThisValue, 3) & " ad in your browser?" & $CRLF & $CRLF & ResultsArray$(ThisValue, 2), %MB_YESNO + %MB_APPLMODAL + %MB_ICONQUESTION, "View it?") IF res = %IDYES THEN pszURL = STRPTR(ResultsArray$(ThisValue, 5)) desktopHandle = GetDesktopWindow() res = ShellExecute(desktopHandle, "open", @pszURL, "", "", %SW_SHOWDEFAULT) 'Load in browser END IF ELSEIF CB.NMCODE = %LVN_COLUMNCLICK THEN 'Click on column header (for sorting) IF SearchRunning <> %FALSE OR DeepScanRunning <> %FALSE THEN MSGBOX "A search is currently active. You need to stop the current scan before sorting the results.", %MB_SYSTEMMODAL OR %MB_ICONERROR, "Error" EXIT SELECT END IF pNMLV = CBLPARAM IF @pNMLV.iSubItem <> -1 THEN 'Column number (starting at 0; with +1... 1=date, 2=title, 3=city, 4=section, 5=fulltext) columnNo = @pNMLV.iSubItem + 1 IF columnNo > 4 THEN EXIT IF 'Don't sort by FullText (no point) 'Perform quick sort of ResultsArray$() Temp$ = "" IF numResults < 2 THEN EXIT IF 'There's nothing to sort: abort LastElement = numResults 'Set up search scope: FirstElement = 1 'From start... NumberOfElements = numResults ' ...to end REDIM QStack(LastElement \ 5 + 10) StackPtr = 0 DO DO Temp$ = LCASE$(ResultsArray$((LastElement \ 2) + (FirstElement \ 2), columnNo)) 'Grab what we're comparing to I = FirstElement J = LastElement DO IF columnNo = 1 THEN 'Date sorts in descending order DO WHILE ResultsArray$(I, columnNo) > Temp$ INCR I LOOP DO WHILE ResultsArray$(J, columnNo) < Temp$ DECR J LOOP ELSE 'Everything else sorts in ascending order (and case-insensitive) DO WHILE LCASE$(ResultsArray$(I, columnNo)) < Temp$ INCR I LOOP DO WHILE LCASE$(ResultsArray$(J, columnNo)) > Temp$ DECR J LOOP END IF IF I > J THEN EXIT DO IF I < J THEN FOR ctr = 1 TO 7 'Swap all sub-elements SWAP ResultsArray$(I, ctr), ResultsArray$(J, ctr) NEXT END IF INCR I DECR J LOOP WHILE I <= J IF I < LastElement THEN QStack(StackPtr) = I QStack(StackPtr + 1) = LastElement INCR StackPtr INCR StackPtr END IF LastElement = J LOOP WHILE FirstElement < LastElement IF StackPtr = 0 THEN EXIT DO DECR StackPtr DECR StackPtr FirstElement = QStack(StackPtr) LastElement = QStack(StackPtr + 1) LOOP 'Sort complete 'Now clear array and rewrite listbox CONTROL SEND tabSearchResults, %IDC_RESULTSLIST, %WM_SETREDRAW, %FALSE, 0 'stop redraw while we rewrite the list LISTVIEW RESET tabSearchResults, %IDC_RESULTSLIST 'erase listview of results FOR ctr = 1 TO numResults 'Re-add them all LISTVIEW INSERT ITEM tabSearchResults, %IDC_RESULTSLIST, ctr, 0, ResultsArray$(ctr, 1) 'Set date column LISTVIEW SET TEXT tabSearchResults, %IDC_RESULTSLIST, ctr, 2, ResultsArray$(ctr, 2) 'Set title colum LISTVIEW SET TEXT tabSearchResults, %IDC_RESULTSLIST, ctr, 3, ResultsArray$(ctr, 3) 'Set city colum LISTVIEW SET TEXT tabSearchResults, %IDC_RESULTSLIST, ctr, 4, ResultsArray$(ctr, 4) 'Set section colum IF LEN(ResultsArray$(ctr, 6)) > 50 THEN 'We have downloaded the full text LISTVIEW SET TEXT tabSearchResults, %IDC_RESULTSLIST, ctr, 5, "Yes" 'Set full text colum END IF LISTVIEW SET USER tabSearchResults, %IDC_RESULTSLIST, ctr, ctr 'Set user data to pointer to full info in ResultsArray$() NEXT CONTROL SEND tabSearchResults, %IDC_RESULTSLIST, %WM_SETREDRAW, %TRUE, 0 'Redraw it now that we're done END IF END IF CASE %IDC_SYSTAB 'Reverse unsaved settings if settings tab is navigated away from IF CBCTLMSG = %BN_CLICKED OR CBCTLMSG = 1 THEN TAB GET SELECT tabSoftwareSettings, %IDC_SYSTAB TO res IF res <> 3 THEN 'Reset the settings if needed CONTROL SET CHECK tabSoftwareSettings, %IDC_CHECK_CL, SearchCraigslist CONTROL SET CHECK tabSoftwareSettings, %IDC_CHECK_KIJIJI, SearchKijiji CONTROL SET CHECK tabSoftwareSettings, %IDC_CHECK_AUTODEEP, AutoDeepScan CONTROL SET CHECK tabSoftwareSettings, %IDC_CHECKZEN, CheckForUpdates CONTROL SET CHECK tabSoftwareSettings, %IDC_DISCARD, DiscardSearches CONTROL SET CHECK tabSoftwareSettings, %IDC_CHECK_SEARCHTITLES, searchTitles CONTROL SET CHECK tabSoftwareSettings, %IDC_CHECK_HASPICTURES, hasPictures CONTROL SET TEXT tabSoftwareSettings, %IDC_MAXRESULTS_BOX, FORMAT$(MaxResults, "0") CONTROL SET TEXT tabSoftwareSettings, %IDC_THREADSRESULTS_BOX, FORMAT$(MaxThreads, "0") CONTROL SET TEXT tabSoftwareSettings, %IDC_THREADS2RESULTS_BOX, FORMAT$(MaxThreads2, "0") CONTROL SET TEXT tabSoftwareSettings, %IDC_USERAGENT_BOX, UserAgent$ 'Note: We could put in a test for changes here and report on it... END IF END IF CASE %IDC_SECTIONS_TREE 'It's in the sections treeview IF CB.NMCODE = %NM_CLICK OR CB.NMCODE = %TVN_KEYDOWN THEN 'Click or Keypress END IF CASE %IDC_CITIESTREE 'It's in the cities treeview itemFlipped = 0 IF CB.NMCODE = %NM_CLICK OR CB.NMCODE = %TVN_KEYDOWN OR CB.NMCODE = %NM_SETFOCUS OR CB.NMCODE = %NM_KILLFOCUS OR CB.NMCODE = %NM_SETCURSOR THEN 'Sooooo sloppy! 'Click or Keypress (keypress can be grabbed with TREEVIEW GET SELECT CB.HNDL, %IDC_CITIESTREE TO tempHandle) doCities() END IF END SELECT END SELECT END FUNCTION 'Turn on main dialog FUNCTION ShowZENCLSHDIALOG(BYVAL hParent AS DWORD) AS LONG LOCAL lRslt AS LONG LOCAL ctr AS LONG, ctr2 AS LONG, isAdded AS LONG LOCAL thisParent AS DWORD, curParent AS DWORD LOCAL dummy AS DWORD LOCAL hDlg AS DWORD LOCAL hFont1 AS DWORD LOCAL hInstance AS LONG 'Note: For now "%WS_MINIMIZEBOX" has been removed so as to allow context-sensitive help DIALOG NEW hParent, "ZenCLSH v" & FORMAT$(%VERSION \ 100) & "." & FORMAT$(%VERSION MOD 100, "00"), _ , , newX - 10, newY - 27, %WS_OVERLAPPED OR %WS_BORDER OR %WS_DLGFRAME OR %WS_SIZEBOX OR %WS_CAPTION OR %WS_SYSMENU OR %WS_CLIPSIBLINGS OR %WS_VISIBLE OR %DS_MODALFRAME OR _ %DS_SETFOREGROUND OR %DS_CENTER OR %DS_CONTEXTHELP OR %DS_3DLOOK OR %DS_NOFAILCREATE OR %DS_SETFONT, %WS_EX_CONTROLPARENT OR %WS_EX_CONTEXTHELP OR %WS_EX_LEFT OR %WS_EX_LTRREADING OR _ %WS_EX_RIGHTSCROLLBAR OR %WS_EX_CONTEXTHELP, TO hDlg mainDialogHandle = hDlg CONTROL ADD LABEL, hDlg, %IDC_SEARCHNAME, "Search name:", 2, 6, 50, 15, %WS_CHILD OR %WS_VISIBLE OR %SS_RIGHT, %WS_EX_LEFT OR %WS_EX_LTRREADING CONTROL ADD TEXTBOX, hDlg, %IDC_SEARCHNAME_BOX, CurrentSearchname$, 55, 5, 150, 15 CONTROL ADD BUTTON, hDlg, %IDC_LOADSEARCH, "Load", 210, 5, 50, 15 CONTROL ADD BUTTON, hDlg, %IDC_SAVESEARCH, "Save", 265, 5, 50, 15 CONTROL ADD TAB, hDlg, %IDC_SYSTAB, "", 5, 30, 375, 230, %TCS_FOCUSONBUTTONDOWN OR %TCS_RAGGEDRIGHT, %WS_EX_LEFT OR %WS_EX_LTRREADING TAB INSERT PAGE hDlg, %IDC_SYSTAB, 1, 0, "Search Results" CALL ShowZENCLSHDIALOGProc TO tabSearchResults CONTROL ADD LISTVIEW, tabSearchResults, %IDC_RESULTSLIST, "Results", 2, 5, 365, 185, %WS_CHILD OR %WS_VISIBLE OR %WS_TABSTOP OR %LVS_REPORT, _ %WS_EX_CLIENTEDGE OR %WS_EX_LEFT OR %WS_EX_RIGHTSCROLLBAR OR %LVS_EX_TWOCLICKACTIVATE OR %LVS_EX_FULLROWSELECT LISTVIEW INSERT COLUMN tabSearchResults, %IDC_RESULTSLIST, 1, "Date", 55, 0 LISTVIEW INSERT COLUMN tabSearchResults, %IDC_RESULTSLIST, 2, "Title", 146, 0 LISTVIEW INSERT COLUMN tabSearchResults, %IDC_RESULTSLIST, 3, "City", 55, 0 LISTVIEW INSERT COLUMN tabSearchResults, %IDC_RESULTSLIST, 4, "Section", 55, 0 LISTVIEW INSERT COLUMN tabSearchResults, %IDC_RESULTSLIST, 5, "Full Text", 50, 0 LISTVIEW SET STYLEXX tabSearchResults, %IDC_RESULTSLIST, %LVS_EX_GRIDLINES OR %LVS_EX_FULLROWSELECT LISTVIEW SET STYLEXX tabSearchResults, %IDC_RESULTSLIST, %LVS_EX_FULLROWSELECT CONTROL ADD BUTTON, tabSearchResults, %IDC_CLEARRESULTS, "Clear", 2, 195, 50, 15 CONTROL ADD BUTTON, tabSearchResults, %IDC_OUTPUT_CSV, "Output CSV", 67, 195, 50, 15 CONTROL ADD BUTTON, tabSearchResults, %IDC_VIEW_HTML, "View HTML", 117, 195, 50, 15 CONTROL ADD BUTTON, tabSearchResults, %IDC_DEEPSCAN, "Deep Scan", 242, 195, 70, 15 CONTROL ADD BUTTON, tabSearchResults, %IDC_GO_SEARCH, "Go!", 317, 195, 50, 15 CONTROL ADD PROGRESSBAR, hDlg, %IDC_PROGRESS, "", 5, 270, 375, 10, %WS_CHILD OR %WS_VISIBLE TAB INSERT PAGE hDlg, %IDC_SYSTAB, 2, 0, "Search Settings" CALL ShowZENCLSHDIALOGProc TO tabSearchSettings CONTROL ADD TREEVIEW, tabSearchSettings, %IDC_CITIESTREE, "", 2, 5, 180, 165, %WS_CHILD OR %WS_VISIBLE OR %WS_TABSTOP OR %TVS_HASBUTTONS OR %TVS_HASLINES OR %TVS_LINESATROOT OR %TVS_SHOWSELALWAYS OR _ %TVS_CHECKBOXES, %WS_EX_CLIENTEDGE OR %WS_EX_LEFT OR %WS_EX_LTRREADING OR %WS_EX_RIGHTSCROLLBAR CONTROL ADD BUTTON, tabSearchSettings, %IDC_CITIESALL, "Select All", 2, 175, 50, 15 CONTROL ADD BUTTON, tabSearchSettings, %IDC_CITIES_CLEARALL, "Clear All", 57, 175, 50, 15 CONTROL ADD TREEVIEW, tabSearchSettings, %IDC_SECTIONS_TREE, "", 187, 5, 180, 165, %WS_CHILD OR %WS_VISIBLE OR %WS_TABSTOP OR %TVS_HASBUTTONS OR %TVS_HASLINES OR %TVS_LINESATROOT OR %TVS_SHOWSELALWAYS OR _ %TVS_CHECKBOXES, %WS_EX_CLIENTEDGE OR %WS_EX_LEFT OR %WS_EX_LTRREADING OR %WS_EX_RIGHTSCROLLBAR CONTROL ADD BUTTON, tabSearchSettings, %IDC_SECTIONSELECTALL, "Select All", 187, 175, 50, 15 CONTROL ADD BUTTON, tabSearchSettings, %IDC_SECTION_CLEARALL, "Clear All", 242, 175, 50, 15 CONTROL ADD LABEL, tabSearchSettings, %IDC_LABEL, "Search terms:", 2, 197, 50, 15, %WS_CHILD OR %WS_VISIBLE OR %SS_RIGHT, %WS_EX_LEFT OR %WS_EX_LTRREADING CONTROL ADD TEXTBOX, tabSearchSettings, %IDC_SEARCHTERMS_BOX, "", 55, 195, 312, 15 TAB INSERT PAGE hDlg, %IDC_SYSTAB, 3, 0, "ZenCLSH Settings" CALL ShowZENCLSHDIALOGProc TO tabSoftwareSettings CONTROL ADD CHECKBOX, tabSoftwareSettings, %IDC_CHECK_CL, "Search Craigslist", 2, 5, 255, 10 CONTROL ADD CHECKBOX, tabSoftwareSettings, %IDC_CHECK_KIJIJI, "Search Kijiji", 2, 15, 255, 10, %WS_DISABLED 'disabled for now CONTROL ADD CHECKBOX, tabSoftwareSettings, %IDC_CHECK_AUTODEEP, "Automatically do a deep scan", 2, 25, 255, 10 CONTROL ADD CHECKBOX, tabSoftwareSettings, %IDC_CHECKZEN, "Check zentastic server for updates", 2, 35, 255, 10 CONTROL ADD CHECKBOX, tabSoftwareSettings, %IDC_DISCARD, "Automatically discard old results after two months", 2, 45, 255, 10 CONTROL ADD LABEL, tabSoftwareSettings, %IDC_MAXLABEL, "Maximum number of search results:", 2, 67, 115, 15, %WS_CHILD OR %WS_VISIBLE OR %SS_RIGHT, %WS_EX_LEFT OR %WS_EX_LTRREADING CONTROL ADD TEXTBOX, tabSoftwareSettings, %IDC_MAXRESULTS_BOX, FORMAT$(MaxResults, "0"), 120, 65, 50, 15 CONTROL ADD LABEL, tabSoftwareSettings, %IDC_THREADSLABEL, "Maximum # of search threads:", 2, 82, 115, 15, %WS_CHILD OR %WS_VISIBLE OR %SS_RIGHT, %WS_EX_LEFT OR %WS_EX_LTRREADING CONTROL ADD TEXTBOX, tabSoftwareSettings, %IDC_THREADSRESULTS_BOX, FORMAT$(MaxThreads, "0"), 120, 80, 50, 15 CONTROL ADD LABEL, tabSoftwareSettings, %IDC_THREADS2LABEL, "Maximum # of deep scan threads:", 2, 97, 115, 15, %WS_CHILD OR %WS_VISIBLE OR %SS_RIGHT, %WS_EX_LEFT OR %WS_EX_LTRREADING CONTROL ADD TEXTBOX, tabSoftwareSettings, %IDC_THREADS2RESULTS_BOX, FORMAT$(MaxThreads2, "0"), 120, 95, 50, 15 CONTROL ADD LABEL, tabSoftwareSettings, %IDC_USERAGENT, "Fake browser user agent:", 2, 112, 115, 15, %WS_CHILD OR %WS_VISIBLE OR %SS_RIGHT, %WS_EX_LEFT OR %WS_EX_LTRREADING CONTROL ADD TEXTBOX, tabSoftwareSettings, %IDC_USERAGENT_BOX, UserAgent$, 120, 110, 200, 15 CONTROL ADD CHECKBOX, tabSoftwareSettings, %IDC_CHECK_SEARCHTITLES, "Only search in the titles of ads", 2, 130, 255, 10 CONTROL ADD CHECKBOX, tabSoftwareSettings, %IDC_CHECK_HASPICTURES, "Only include ads with images in the search results", 2, 140, 255, 10 CONTROL ADD BUTTON, tabSoftwareSettings, %IDC_APPLY_SETTINGS, "Apply", 57, 195, 50, 15 CONTROL ADD BUTTON, tabSoftwareSettings, %IDC_SETTINGS_CANCEL, "Cancel", 2, 195, 50, 15 CONTROL ADD BUTTON, tabSoftwareSettings, %IDC_VISITWEB, "About...", 307, 195, 60, 15 TAB INSERT PAGE hDlg, %IDC_SYSTAB, 4, 0, "Diagnostics" CALL ShowZENCLSHDIALOGProc TO tabDiagnostics FOR ctr = 1 TO 12 CONTROL ADD LABEL, tabDiagnostics, %IDC_STHREAD + ctr, FORMAT$(ctr, "00") & ": inactive", 12, 5 + (ctr * 10), 255, 10 NEXT FOR ctr = 1 TO 12 CONTROL ADD LABEL, tabDiagnostics, %IDC_DTHREAD + ctr, "DS" & FORMAT$(ctr, "00") & ": inactive", 12, 135 + (ctr * 10), 255, 10 NEXT hFont1 = PBFormsMakeFont("MS Sans Serif", 8, 700, %FALSE, %FALSE, %FALSE, %ANSI_CHARSET) CONTROL SEND tabSearchResults, %IDC_GO_SEARCH, %WM_SETFONT, hFont1, 0 'Add sections to Section Treeview REDIM SectionArrayHandle(1 TO 500) AS DWORD FOR ctr = 1 TO numSections 'If section=supersection, add it to the top of the tree, and if it's not the same, add it as a child of the most recent supersection IF SectionList$(ctr, 1) = SectionList$(ctr, 2) THEN TREEVIEW INSERT ITEM tabSearchSettings, %IDC_SECTIONS_TREE, 0, %TVI_SORT, 0, 0, SectionList$(ctr, 3) TO curParent SectionArrayHandle(ctr) = curParent ELSE TREEVIEW INSERT ITEM tabSearchSettings, %IDC_SECTIONS_TREE, curParent, %TVI_SORT, 0, 0, SectionList$(ctr, 3) TO SectionArrayHandle(ctr) END IF 'Check if needed IF SectionList$(ctr, 4) <> "0" AND SectionList$(ctr, 4) <> "" THEN TREEVIEW SET CHECK tabSearchSettings, %IDC_SECTIONS_TREE, SectionArrayHandle(ctr), %TRUE END IF NEXT 'Arrays for the city picker REDIM RegionArray$(1 TO 10) REDIM RegionArrayHandle(1 TO 10) AS DWORD REDIM RegionArrayStatus(1 TO 10) AS LONG REDIM CountryArray$(1 TO 200) REDIM CountryArrayHandle(1 TO 200) AS DWORD REDIM CountryArrayStatus(1 TO 200) AS LONG REDIM CityArrayHandle(1 TO 1000) AS DWORD REDIM CityArrayStatus(1 TO 1000) AS LONG tempNumRegions = 0 tempNumCountries = 0 'Add regions to city treeview FOR ctr = 1 TO numCities 'Is it already added? isAdded = 0 FOR ctr2 = 1 TO tempNumRegions IF RegionArray$(ctr2) = CityList$(ctr, 1) THEN isAdded = 1 EXIT FOR END IF NEXT 'Add it IF isAdded = 0 THEN INCR tempNumRegions RegionArray$(tempNumRegions) = CityList$(ctr, 1) RegionArrayStatus(tempNumRegions) = %FALSE TREEVIEW INSERT ITEM tabSearchSettings, %IDC_CITIESTREE, 0, %TVI_SORT, 0, 0, RegionArray$(tempNumRegions) TO RegionArrayHandle(tempNumRegions) END IF NEXT 'Add countries to city treeview FOR ctr = 1 TO numCities 'Is it already added? isAdded = 0 FOR ctr2 = 1 TO tempNumCountries IF CountryArray$(ctr2) = CityList$(ctr, 3) THEN isAdded = 1 EXIT FOR END IF NEXT IF isAdded = 0 THEN 'Find parent region FOR ctr2 = 1 TO tempNumRegions IF RegionArray$(ctr2) = CityList$(ctr, 1) THEN thisParent = RegionArrayHandle(ctr2) EXIT FOR END IF NEXT 'Add it INCR tempNumCountries CountryArray$(tempNumCountries) = CityList$(ctr, 3) CountryArrayStatus(tempNumCountries) = %FALSE TREEVIEW INSERT ITEM tabSearchSettings, %IDC_CITIESTREE, thisParent, %TVI_SORT, 0, 0, CountryArray$(tempNumCountries) TO CountryArrayHandle(tempNumCountries) END IF NEXT 'Add cities to city treeview FOR ctr = 1 TO numCities 'Find parent country FOR ctr2 = 1 TO tempNumCountries IF CountryArray$(ctr2) = CityList$(ctr, 3) THEN thisParent = CountryArrayHandle(ctr2) EXIT FOR END IF NEXT 'Add it IF CityList$(ctr, 4) = "Default" THEN 'Just use the country CityArrayHandle(ctr) = thisParent IF CityList$(ctr, 6) <> "0" AND CityList$(ctr, 6) <> "" THEN TREEVIEW SET CHECK tabSearchSettings, %IDC_CITIESTREE, thisParent, %TRUE CityArrayStatus(ctr) = ABS(%TRUE) ELSE CityArrayStatus(ctr) = %FALSE END IF ELSE 'Normal add TREEVIEW INSERT ITEM tabSearchSettings, %IDC_CITIESTREE, thisParent, %TVI_SORT, 0, 0, CityList$(ctr, 4) TO tempHandle CityArrayHandle(ctr) = tempHandle CityArrayStatus(ctr) = %FALSE IF CityList$(ctr, 6) <> "0" AND CityList$(ctr, 6) <> "" THEN TREEVIEW SET CHECK tabSearchSettings, %IDC_CITIESTREE, tempHandle, %TRUE CityArrayStatus(ctr) = ABS(%TRUE) ELSE CityArrayStatus(ctr) = %FALSE END IF END IF NEXT 'Initialize settings window (other than the ones that are set on create above) CONTROL SET CHECK tabSoftwareSettings, %IDC_CHECK_CL, SearchCraigslist CONTROL SET CHECK tabSoftwareSettings, %IDC_CHECK_KIJIJI, SearchKijiji CONTROL SET CHECK tabSoftwareSettings, %IDC_CHECK_AUTODEEP, AutoDeepScan CONTROL SET CHECK tabSoftwareSettings, %IDC_CHECKZEN, CheckForUpdates CONTROL SET CHECK tabSoftwareSettings, %IDC_DISCARD, DiscardSearches CONTROL SET CHECK tabSoftwareSettings, %IDC_CHECK_SEARCHTITLES, searchTitles CONTROL SET CHECK tabSoftwareSettings, %IDC_CHECK_HASPICTURES, hasPictures 'Set up and clear progress bar PROGRESSBAR SET RANGE hDlg, %IDC_PROGRESS, 0, 1000 'bar goes from 0 to 1000 PROGRESSBAR SET POS hDlg, %IDC_PROGRESS, 0 'Load current search (but don't warn if it's not there) LoadSearch %TRUE 'Check to see if software has been updated IF CheckForUpdates <> %FALSE THEN THREAD CREATE CheckVer(dummy) TO vThread??? END IF 'Turn dialog on, with pretty icon hInstance = GetModuleHandle("") DIALOG SEND hDlg, %WM_SETICON, %ICON_BIG, LoadIcon(hInstance, "HLBICON") DIALOG SHOW MODAL hDlg, CALL ShowZENCLSHDIALOGProc TO lRslt 'We're outta here DeleteObject hFont1 FUNCTION = lRslt END FUNCTION FUNCTION SearchMe(BYVAL searchNo AS DWORD) AS DWORD 'Perform a search LOCAL theItem$(), theLink$(), theDate$(), theDescription$() LOCAL tryCtr AS LONG, numItems AS LONG, ctr4 AS LONG, dummy2 AS LONG, thisMatch AS LONG, ctr AS LONG, foundIt AS LONG, timeDelay AS LONG DIM theItem$(1 TO MaxResults), theLink$(1 TO MaxResults), theDate$(1 TO MaxResults), theDescription$(1 TO MaxResults), dummy AS DWORD tryCtr = 0 keeptrying: CONTROL SET TEXT tabDiagnostics, %IDC_STHREAD + searchNo, FORMAT$(searchNo, "00") & ": " & sTerm$(searchNo) & " in " & sPrettySection$(searchNo) & " in " & sPrettyCity$(searchNo) IF SearchAbort <> %FALSE THEN GOTO abortfinish 'Watch for an abort signal 'Grab RSS search page numItems = GetCraigsRSS(sTerm$(searchNo), sCity$(searchNo), sSection$(searchNo), theItem$(), theLink$(), theDate$(), theDescription$()) IF numItems = -1 THEN 'Did we get a valid result? If not, retry a few times INCR tryCtr IF tryCtr > 4 THEN 'Tried five times with no luck IF connectFailUp = %FALSE THEN 'Don't say this over and over (because of multi-threading) connectFailUp = %TRUE MSGBOX "I'm having trouble connecting to the search for " & sTerm$(searchNo) & " on the server " & sCity$(searchNo) & " in the section " & sSection$(searchNo) & ". Aborting search, sorry.", %MB_SYSTEMMODAL OR %MB_ICONERROR, "Error" connectFailUp = %FALSE END IF GOTO abortfinish END IF timeDelay = 25 + INT(RND * 11) 'Pause for a while FOR ctr = timeDelay TO 1 STEP -1 '25 - 35 second pause on fail (random so as to put out of sync) CONTROL SET TEXT tabDiagnostics, %IDC_STHREAD + searchNo, FORMAT$(searchNo, "00") & ": Connect problem (" & sPrettyCity$(searchNo) & ")... Restarting in " & FORMAT$(ctr) FOR ctr4 = 1 TO 10 IF SearchAbort <> %FALSE THEN GOTO abortfinish 'Watch for an abort signal SLEEP 100 NEXT NEXT GOTO keepTrying END IF FOR ctr4 = 1 TO numItems 'Go through all found items 'Add results to results table and results array IF TRIM$(theItem$(ctr4)) = "" OR TRIM$(theLink$(ctr4)) = "" THEN ITERATE FOR 'Bad item foundIt = 0 FOR ctr = 1 TO numResults IF INSTR(ResultsArray$(ctr, 5), theLink$(ctr4)) > 0 THEN foundIt = ctr EXIT FOR END IF NEXT IF foundIt = 0 THEN 'Write to array DO IF lockListview = %FALSE THEN EXIT DO 'Wait for listView to be unlocked LOOP THREAD SET PRIORITY sThread???(searchNo), %THREAD_PRIORITY_TIME_CRITICAL 'Avoid collisions with other threads by going to high priority and... EnterCriticalSection gCS ' ...entering a critical section. lockListview = %TRUE 'Lock listview INCR numResults IF numResults > maxResults THEN DECR numResults MSGBOX "The results list has overflowed (" & FORMAT$(numResults, "0,") & " elements). Please increase max results in the settings.", %MB_SYSTEMMODAL OR %MB_ICONERROR, "Error" EXIT FOR END IF thisMatch = numResults ResultsArray$(thisMatch, 1) = TRIM$(theDate$(ctr4)) 'Date ResultsArray$(thisMatch, 2) = TRIM$(theItem$(ctr4)) 'Title ResultsArray$(thisMatch, 3) = sPrettyCity$(searchNo) 'City ResultsArray$(thisMatch, 4) = sPrettySection$(searchNo) 'Section ResultsArray$(thisMatch, 5) = TRIM$(theLink$(ctr4)) 'Link ResultsArray$(thisMatch, 6) = "" 'Full text (obviously not done yet) ResultsArray$(thisMatch, 7) = "Yes" 'This element/ad is newly downloaded this session 'Add to top of listbox LISTVIEW INSERT ITEM tabSearchResults, %IDC_RESULTSLIST, 1, 0, ResultsArray$(numResults, 1) 'Set date column LISTVIEW SET TEXT tabSearchResults, %IDC_RESULTSLIST, 1, 2, ResultsArray$(numResults, 2) 'Set title colum LISTVIEW SET TEXT tabSearchResults, %IDC_RESULTSLIST, 1, 3, ResultsArray$(numResults, 3) 'Set city colum LISTVIEW SET TEXT tabSearchResults, %IDC_RESULTSLIST, 1, 4, ResultsArray$(numResults, 4) 'Set section colum LISTVIEW SET USER tabSearchResults, %IDC_RESULTSLIST, 1, numResults 'Set user data to pointer to full info in ResultsArray$() LeaveCriticalSection gCS 'Exit critical section and... THREAD SET PRIORITY sThread???(searchNo), %THREAD_PRIORITY_NORMAL ' ...resume normal priority. lockListview = %FALSE 'Unlock listview UnsavedResults = %TRUE 'Make sure we don't accidentally toss these matches 'Are we automatically doing a deep scan or has one been manually turned on? IF autoDeepScan = %FALSE AND DeepScanRunning = %FALSE THEN ITERATE FOR 'Skip the rest of this loop if there's no need for deep scanning END IF 'If a deep scan is not active, turn on a thread THREAD CREATE doLaunchDeep(dummy) TO lThread??? END IF NEXT abortfinish: CONTROL SET TEXT tabDiagnostics, %IDC_STHREAD + searchNo, FORMAT$(searchNo, "00") & ": inactive" sTerm$(searchNo) = "" sCity$(searchNo) = "" sSection$(searchNo) = "" END FUNCTION FUNCTION CheckVer(BYVAL dummy AS DWORD) AS DWORD 'Has the software been updated? (Code borrowed from old version) LOCAL buffer$, thisVer$, VerCode AS LONG, temp$ LOCAL desktopHandle AS DWORD LOCAL pszURL AS ASCIIZ PTR, res AS LONG buffer$ = httpGrab$($WEBPAGESITE, $WEBPAGEPAGE, %TRUE) 'Grab the file thisVer$ = Between$(buffer$, "", "") 'Try and find the version text IF thisVer$ <> "" THEN VerCode = VAL(thisVer$) * 100 IF VerCode > %VERSION THEN res = MSGBOX("A new version exists (" & FORMAT$(VerCode / 100, "0.00") & "). Would you like to visit the software's homepage?", %MB_ICONQUESTION + %MB_YESNO + %MB_APPLMODAL, "Update Available") IF res = %IDYES THEN temp$ = "http://" & $WEBPAGESITE & $WEBPAGEPAGE pszURL = STRPTR(temp$) desktopHandle = GetDesktopWindow() res = ShellExecute(desktopHandle, "open", @pszURL, "", "", %SW_SHOWDEFAULT) END IF END IF END IF END FUNCTION FUNCTION Between$(inString$, starter$, finisher$) 'find the text between two markers... this is a very useful function! LOCAL startat AS LONG, endat AS LONG startat = INSTR(inString$, starter$) IF startat = 0 THEN 'Is there a valid beginning code? FUNCTION = "" ELSE startat = startat + LEN(starter$) endat = INSTR(startat, inString$, finisher$) IF endat = 0 THEN 'Is there an end code? FUNCTION = MID$(inString$, startat) ELSE 'If not, just output everything from the beginning to the end of the string FUNCTION = MID$(inString$, startat, endat - startat) END IF END IF END FUNCTION FUNCTION httpGrab$(site$, file$, allowredirect AS LONG) 'Simply grabs a file via HTTP GET 'Multiple calls of this function can be run concurrently to optimize speed LOCAL buffer$, res$ LOCAL newSite$, newFile$, newURI$ LOCAL ff AS LONG ff = FREEFILE TCP OPEN PORT 80 AT site$ AS #ff TIMEOUT 5500 TCP PRINT #ff, "GET " & file$ & " HTTP/1.0" TCP PRINT #ff, "Host: " & site$ TCP PRINT #ff, "User-Agent: " & UserAgent$ TCP PRINT #ff, "Accept-Language: en-us" 'Without this craigslist returns a failed document because it tries to auto-detect language TCP PRINT #ff, "" res$ = "" DO 'Suck in the http stream buffer$ = "" TCP RECV #ff, 4096, buffer res$ = res$ & buffer$ LOOP WHILE LEN(buffer$) TCP CLOSE #ff 'Check if we need to reload a redirect IF INSTR(res$, "302 Found") > 0 AND allowredirect = %TRUE THEN '302s will occur on some URLs that have been moved to a country-based domain newURI$ = Between$(res$, "Location: http://", $CRLF) newSite$ = MID$(newURI$, 1, INSTR(newURI$, "/") - 1) newFile$ = MID$(newURI$, INSTR(newURI$, "/")) res$ = httpGrab$(newSite$, newFile$, %FALSE) 'Don't redirect, in case we get stuck in a loop! END IF 'And we're done! FUNCTION = res$ END FUNCTION FUNCTION DoHarvest(BYVAL dummy AS DWORD) AS DWORD 'Harvesting (of RSS search results) function LOCAL ctr AS LONG, ctr2 AS LONG, ctr3 AS LONG, ctr4 AS LONG LOCAL totalSearches AS LONG, stageAt AS LONG, dummy2 AS LONG LOCAL thisCity$, thisSection$, thisTerm$, thisDeepURL$ LOCAL tryCtr AS LONG, numItems AS LONG, perMil AS LONG LOCAL thisPrettyCity$, thisPrettySection$, foundThread AS LONG LOCAL numSearchCities AS LONG, numSearchSections AS LONG, numSearchTerms AS LONG numSearchCities = PARSECOUNT(doCityList$) numSearchSections = PARSECOUNT(doSectionList$) numSearchTerms = PARSECOUNT(doTermList$) totalSearches = numSearchCities * numSearchSections * numSearchTerms FOR ctr = 1 TO numSearchCities 'Loop through all the cities thisCity$ = PARSE$(doCityList$, ctr) 'Pretty up the city name thisPrettyCity$ = "" FOR ctr4 = 1 TO numCities IF CityList$(ctr4, 5) = thisCity$ THEN thisPrettyCity$ = CityList$(ctr4, 4) IF CityList$(ctr4, 4) = "Default" THEN thisPrettyCity$ = CityList$(ctr4, 3) EXIT FOR END IF END IF NEXT FOR ctr2 = 1 TO numSearchSections 'Loop through all the sections thisSection$ = PARSE$(doSectionList$, ctr2) 'Pretty up the section name thisPrettySection$ = "" FOR ctr4 = 1 TO numSections IF SectionList$(ctr4, 2) = thisSection$ THEN thisPrettySection$ = SectionList$(ctr4, 3) EXIT FOR END IF NEXT FOR ctr3 = 1 TO numSearchTerms 'Loop through all the terms IF SearchAbort <> %FALSE THEN GOTO abortfinish 'Watch for an abort signal 'Set progress bar stageAt = (ctr3 - 1) + ((ctr2 - 1) * numSearchTerms) + ((ctr - 1) * numSearchTerms * numSearchSections) perMil = (stageAt * 1000) / totalSearches PROGRESSBAR SET POS mainDialogHandle, %IDC_PROGRESS, perMil 'Set progress bar as is relevant 'Set up search terms thisTerm$ = PARSE$(doTermList$, ",", ctr3) 'Lauch search thread DO foundThread = 0 FOR ctr4 = 1 TO MaxThreads IF sTerm$(ctr4) = "" THEN 'available! sTerm$(ctr4) = thisTerm$ sCity$(ctr4) = thisCity$ sSection$(ctr4) = thisSection$ sPrettyCity$(ctr4) = thisPrettyCity$ sPrettySection$(ctr4) = thisPrettySection$ THREAD CREATE SearchMe(ctr4) TO sThread???(ctr4) 'Start searching foundThread = ctr4 EXIT FOR END IF NEXT IF foundThread > 0 THEN EXIT DO SLEEP 150 'If threads are full, pause for a tiny moment LOOP NEXT NEXT NEXT PROGRESSBAR SET POS mainDialogHandle, %IDC_PROGRESS, 1000 'Fill progress bar since this is a true finish abortfinish: DO foundThread = 0 FOR ctr4 = 1 TO MaxThreads IF sTerm$(ctr4) <> "" THEN 'Thread is active foundThread = ctr4 END IF NEXT IF foundThread = 0 THEN EXIT DO SLEEP 150 'Wait for all threads to finish LOOP DIALOG SET TEXT mainDialogHandle, "ZenCLSH v" & FORMAT$(%VERSION \ 100) & "." & FORMAT$(%VERSION MOD 100, "00") 'If we finish or get an abort, reset everything as it should be 'Switch controls back to what they should be CONTROL SET TEXT tabSearchResults, %IDC_GO_SEARCH, "Go!" CONTROL ENABLE mainDialogHandle, %IDC_LOADSEARCH CONTROL ENABLE mainDialogHandle, %IDC_SEARCHNAME_BOX CONTROL ENABLE tabSearchSettings, %IDC_CITIESTREE CONTROL ENABLE tabSearchSettings, %IDC_CITIESALL CONTROL ENABLE tabSearchSettings, %IDC_CITIES_CLEARALL CONTROL ENABLE tabSearchSettings, %IDC_SECTIONS_TREE CONTROL ENABLE tabSearchSettings, %IDC_SECTIONSELECTALL CONTROL ENABLE tabSearchSettings, %IDC_SECTION_CLEARALL CONTROL ENABLE tabSearchSettings, %IDC_SEARCHTERMS_BOX CONTROL ENABLE tabSoftwareSettings, %IDC_CHECK_CL 'CONTROL enable tabSoftwareSettings, %IDC_CHECK_KIJIJI 'Not currently active CONTROL ENABLE tabSoftwareSettings, %IDC_CHECK_AUTODEEP CONTROL ENABLE tabSoftwareSettings, %IDC_SETTINGS_CANCEL CONTROL ENABLE tabSoftwareSettings, %IDC_APPLY_SETTINGS SearchRunning = %FALSE END FUNCTION FUNCTION doLaunchDeep(BYVAL dummy AS DWORD) AS DWORD 'This function just launches a deep scan LOCAL ctr AS LONG FOR ctr = 1 TO MaxThreads2 IF dActive(ctr) = %FALSE THEN THREAD CREATE doDeepScan(ctr) TO dThread???(ctr) 'Begin deep scan thread SLEEP 1000 'Pause one second between activations END IF NEXT END FUNCTION FUNCTION doDeepScan(BYVAL whichScan AS DWORD) AS DWORD 'Execute a deep scan (assumes list is pre-saturated) LOCAL thisScan$, thisData$, thisSite$, thisFile$, tempEmail$, notAllDone AS LONG, timeDelay AS LONG, foundLost AS LONG LOCAL failCtr AS LONG, foundAt AS LONG, foundUser AS LONG, ctr AS LONG, userbody AS LONG, ctr4 AS LONG DeepScanRunning = %TRUE 'Just in case, turn it on (probably redundant) dActive(whichScan) = 1 CONTROL SET TEXT tabSearchResults, %IDC_DEEPSCAN, "Abort DS" 'Make sure button says "abort" DO foundLost = 0 FOR ctr = 1 TO numResults 'Find the first un-downloaded entry IF ResultsArray$(ctr, 6) = "" THEN 'Fulltext is blank ResultsArray$(ctr, 6) = "Pending" 'Set to "pending" so other threads don't do it foundLost = ctr EXIT FOR END IF NEXT IF foundLost = 0 THEN EXIT DO 'Finished! thisScan$ = ResultsArray$(foundLost, 5) 'Grab URL to download IF thisScan$ = "" THEN 'This should never happen; could be a corrupt file ResultsArray$(ctr, 5) = "Error" ResultsArray$(ctr, 6) = "Error" EXIT DO END IF CONTROL SET TEXT tabDiagnostics, %IDC_DTHREAD + whichScan, "DS" & FORMAT$(whichScan, "00") & ": " & thisScan$ 'Update diagnostic message 'Pull out site and file IF LEFT$(thisScan$, 7) = "http://" THEN thisScan$ = MID$(thisScan$, 8) 'trim leading "http://" thisSite$ = LEFT$(thisScan$, INSTR(thisScan$, "/") - 1) 'Extract site thisFile$ = MID$(thisScan$, INSTR(thisScan$, "/")) 'Extract file failCtr = 0 'Reset fail counter tryDSagain: CONTROL SET TEXT tabDiagnostics, %IDC_DTHREAD + whichScan, "DS" & FORMAT$(whichScan, "00") & ": http://" & thisScan$ 'Update diagnostic message 'Check for abort IF DeepScanAbort <> %FALSE THEN GOTO abortfinishDS thisData$ = httpGrab$(thisSite$, thisFile$, %TRUE) 'Download this ad 'Try and crop data down userbody = INSTR(thisData$, "
") IF userbody > 0 THEN '(1) Pull out email address tempEmail$ = "Could not find a valid email address." userbody = INSTR(thisData$, " 0 THEN tempEmail$ = MID$(thisData$, userbody) userbody = INSTR(tempEmail$, "") IF userbody > 0 THEN tempEmail$ = LEFT$(tempEmail$, userbody + 3) END IF END IF '(2) Crop to just "userbody" div userbody = INSTR(thisData$, "
") thisData$ = MID$(thisData$, userbody + 19) userbody = INSTR(thisData$, "
") IF userbody > 0 THEN thisData$ = LEFT$(thisData$, userbody - 1) END IF '(3) re-assemble thisData$ = tempEmail$ & "

" & $CRLF & thisData$ END IF IF LEN(thisData$) < 50 THEN 'Must be crap INCR failCtr IF failCtr > 4 THEN MSGBOX "I'm having trouble downloading " & thisFile$ & " from the server " & thisSite$ & ". Aborting deep scan, sorry.", %MB_SYSTEMMODAL OR %MB_ICONERROR, "Error" ResultsArray$(foundLost, 6) = "" 'We'll need to redo it later GOTO abortfinishDS END IF timeDelay = 25 + INT(RND * 11) 'Pause for a while FOR ctr = timeDelay TO 1 STEP -1 '25 - 35 second pause on fail (random so as to put out of sync) CONTROL SET TEXT tabDiagnostics, %IDC_DTHREAD + whichScan, "DS" & FORMAT$(whichScan, "00") & ": Connect problem (" & thisSite$ & ")... Restarting in " & FORMAT$(ctr) FOR ctr4 = 1 TO 10 IF DeepScanAbort <> %FALSE THEN GOTO abortfinishDS 'Watch for an abort signal SLEEP 100 NEXT NEXT GOTO tryDSagain END IF 'Add deep scan data to results array and update results listview to reflect we have the data foundAt = 0 FOR ctr = 1 TO numResults 'Try and find where in the array this goes IF INSTR(ResultsArray$(ctr, 5), thisScan$) > 0 THEN foundAt = ctr EXIT FOR END IF NEXT IF foundAt > 0 THEN ResultsArray$(foundAt, 6) = thisData$ 'Put downloaded data into array FOR ctr = 1 TO numResults LISTVIEW GET USER tabSearchResults, %IDC_RESULTSLIST, ctr TO foundUser IF foundUser = foundAt THEN 'We found the right element LISTVIEW SET TEXT tabSearchResults, %IDC_RESULTSLIST, ctr, 5, "Yes" 'Indicate we have long text UnsavedResults = %TRUE EXIT FOR END IF NEXT END IF LOOP abortfinishDS: 'Deep scan is finished CONTROL SET TEXT tabDiagnostics, %IDC_DTHREAD + whichScan, "DS" & FORMAT$(whichScan, "00") & ": inactive" 'Deactivate diagnostic message dActive(whichScan) = 0 'Label this thread as inactive notAllDone = 0 FOR ctr = 1 TO 12 'See if other threads are still active IF dActive(ctr) = 1 THEN notAllDone = 1 EXIT FOR END IF NEXT IF notAllDone = 0 THEN CONTROL SET TEXT tabSearchResults, %IDC_DEEPSCAN, "Deep Scan" 'Button changes back to normal DeepScanRunning = %FALSE 'Allow new deep scans to be done END IF END FUNCTION FUNCTION GetCraigsRSS(Query$, City$, Section$, Item$(), Link$(), itemDate$(), Description$()) AS LONG 'Harvest a Craigslist RSS feed 'Query$ is what to search for -- ie. '"kit car"' or 'wheelchair' 'City$ is the city (URL reference) -- ie. 'toronto' or 'flint' 'Section$ is the section of the site to search -- ie. 'sss' for all, 'car' for cars, 'clt' for collectibles 'Item$(), Link$(), Description$() return the results 'Function returns the number of results LOCAL theSite$, theURI$, rssFeed$, numItems AS LONG, ctr AS LONG, sectInsert$, tempDate$ IF INSTR(City$, ".craigslist") > 0 THEN IF VAL(Section$) > 0 THEN EXIT FUNCTION 'trying to search craigslist for a kijiji section theSite$ = City$ 'assume full URL given theURI$ = "/search/" & Section$ & "?query=" & toPut$(Query$) & "&format=rss" IF searchTitles <> %FALSE THEN theURI$ = theURI$ & "&srchType=T" IF hasPictures <> %FALSE THEN theURI$ = theURI$ & "&hasPic=1" ELSEIF INSTR(City$, ".kijiji") > 0 THEN theSite$ = City$ 'assume full URL given sectInsert$ = "CatId=" & Section$ & "&" IF Section$ = "sss" OR Section$ = "" THEN sectInsert$ = "" END IF theURI$ = "/f-SearchAdRss?" & sectInsert$ & "Keyword=" & toPut$(Query$) ELSE IF VAL(Section$) > 0 THEN EXIT FUNCTION 'trying to search craigslist for a kijiji section theSite$ = City$ & ".craigslist.org" theURI$ = "/search/" & Section$ & "?query=" & toPut$(Query$) & "&format=rss" IF searchTitles <> %FALSE THEN theURI$ = theURI$ & "&srchType=T" IF hasPictures <> %FALSE THEN theURI$ = theURI$ & "&hasPic=1" END IF IF LEFT$(theSite$, 7) = "http://" THEN theSite$ = MID$(theSite$, 8) 'Trim leading http, if included IF RIGHT$(theSite$, 1) = "/" THEN theSite$ = LEFT$(theSite$, LEN(theSite$) - 1) 'Trim trailing "/", if needed rssFeed$ = httpGrab$(theSite$, theURI$, %TRUE) IF LEN(rssFeed$) > 0 THEN IF INSTR(theSite$, "craigslist") > 0 THEN rssFeed$ = MID$(rssFeed$, INSTR(rssFeed$, "")) 'chop off header, don't need it numItems = TALLY(rssFeed$, "") IF numItems > 0 THEN FOR ctr = 1 TO numItems Item$(ctr) = Between$(rssFeed$, "<title>", "") IF LEFT$(Item$(ctr), 9) = "") END IF itemDate$(ctr) = LEFT$(Between$(rssFeed$, "", ""), 10) Description$(ctr) = Between$(rssFeed$, "", "") IF LEFT$(Description$(ctr), 9) = "") END IF REPLACE "&" WITH "&" IN Description$(ctr) Link$(ctr) = Between$(rssFeed$, "", "") rssFeed$ = MID$(rssFeed$, INSTR(rssFeed$, "") + 7) 'chop off what we've done NEXT FUNCTION = numItems ELSE FUNCTION = 0 'no matches END IF ELSEIF INSTR(theSite$, "kijiji") > 0 THEN rssFeed$ = MID$(rssFeed$, INSTR(rssFeed$, "")) numItems = TALLY(rssFeed$, "") IF numItems > 0 THEN FOR ctr = 1 TO numItems Item$(ctr) = Between$(rssFeed$, "<title>", "") IF LEFT$(Item$(ctr), 9) = "") END IF tempDate$ = Between$(rssFeed$, "", "") 'Format is "Sat, 26 Jul 2008 02:41:00 GMT" which needs to be YYYY-MM-DD itemDate$(ctr) = PARSE$(tempDate$, " ", 4) & "-" & FORMAT$(numFromMonth(PARSE$(tempDate$, " ", 3)), "00") & "-" & FORMAT$(VAL(PARSE$(tempDate$, " ", 2)), "00") Description$(ctr) = Between$(rssFeed$, "", "") IF LEFT$(Description$(ctr), 9) = "") END IF Link$(ctr) = Between$(rssFeed$, "", "") rssFeed$ = MID$(rssFeed$, INSTR(rssFeed$, "") + 7) 'chop off what we've done NEXT FUNCTION = numItems ELSE FUNCTION = 0 END IF END IF ELSE FUNCTION = -1 'error, probably http error END IF END FUNCTION FUNCTION CSVsafe$(inString$) 'Makes sure a string is safe to use in a CSV file LOCAL temp$ temp$ = inString$ REPLACE "\" WITH "\\" IN temp$ ' \ becomes \\ (escpae character) REPLACE """" WITH "\""" IN temp$ ' " becomes \" (no double quotes allowed unless escaped) REPLACE $CRLF WITH "\n" IN temp$ ' CRLF becomes \n (keep it all one line) REPLACE $CR WITH "\n" IN temp$ ' ...as above (\n) in case of non-typical line breaks REPLACE $LF WITH "\n" IN temp$ ' ...as above REPLACE $FF WITH "\n" IN temp$ ' ...as above FUNCTION = temp$ END FUNCTION FUNCTION ParseCSV$(inString$, termNo AS LONG) 'Pull out a string from a CSV and reverse the encoding LOCAL temp$, temp2$ temp$ = inString$ REPLACE "\""" WITH CHR$(255) IN temp$ 'Deal with quotes (stage one) temp2$ = PARSE$(temp$, termNo) 'Extract specific term REPLACE CHR$(255) WITH """" IN temp2$ 'Restore quotes (stage two) REPLACE "\\" WITH "\" IN temp2$ 'restore backslashes REPLACE "\n" WITH $CRLF IN temp2$ 'restore line breaks FUNCTION = temp2$ END FUNCTION SUB ShowPopupHelp(sText$) 'Pops up a help window for the context-sensitive help system LOCAL hWnd AS LONG LOCAL thehh_popup AS HH_POPUP GetCursorPos thehh_popup.pt hWnd = WindowFromPoint(thehh_popup.pt.X, thehh_popup.pt.Y) thehh_popup.cbStruct = LEN(thehh_popup) thehh_popup.hinst = GetWindowLong(hWnd, %GWL_HINSTANCE) thehh_popup.idString = 0 thehh_popup.pszText = STRPTR(sText$) thehh_popup.pt.x = thehh_popup.pt.x + 25 thehh_popup.pt.y = thehh_popup.pt.y + 25 thehh_popup.clrForeground = 0 thehh_popup.clrBackground = &HE0FFFF thehh_popup.rcMargins.nRight = -1 thehh_popup.rcMargins.nLeft = -1 thehh_popup.rcMargins.nTop = -1 thehh_popup.rcMargins.nBottom = -1 HtmlHelp hWnd, BYVAL %NULL, %HH_DISPLAY_TEXT_POPUP, BYVAL VARPTR(thehh_popup) END SUB FUNCTION numFromMonth(numName$) AS LONG 'convert text month to numeric LOCAL result AS LONG SELECT CASE LCASE$(LEFT$(numName$, 3)) CASE "jan" result = 1 CASE "feb" result = 2 CASE "mar" result = 3 CASE "apr" result = 4 CASE "may" result = 5 CASE "jun" result = 6 CASE "jul" result = 7 CASE "aug" result = 8 CASE "sep" result = 9 CASE "oct" result = 10 CASE "nov" result = 11 CASE "dec" result = 12 END SELECT FUNCTION = result END FUNCTION FUNCTION doDownload((BYVAL dummy AS DWORD) AS DWORD 'Download city/section settings (threaded to avoid crashes) LOCAL temp$, OutputFile$, windowText$, thisCountry$, geoSite$, linkURL$, linkName$ LOCAL thisSectionCode$, thisSectionName$, sectSite$, thisSectCode$, thisSectName$, thisAlpha2$ LOCAL ctr AS LONG, errCount AS LONG, startCode AS LONG, tLen AS LONG, numCountries AS LONG, linkStart AS LONG windowText$ = "" CONTROL DISABLE dlWindowHandle, %IDC_BEGIN_DOWNLOAD CONTROL SET TEXT dlWindowHandle, %IDC_BEGIN_DOWNLOAD, "Grabbing Sections" '=======DOWNLOAD SECTION DATA BEGIN======= OutputFile$ = """MasterSectionCode"",""SectionCode"",""SectionName""" & $CRLF FOR ctr = 1 TO 9 SELECT CASE ctr CASE 1 thisSectionCode$ = "ccc" thisSectionName$ = "Community (All)" CASE 2 thisSectionCode$ = "eee" thisSectionName$ = "Events (All)" CASE 3 thisSectionCode$ = "ggg" thisSectionName$ = "Gigs (All)" CASE 4 thisSectionCode$ = "hhh" thisSectionName$ = "Housing (All)" CASE 5 thisSectionCode$ = "jjj" thisSectionName$ = "Jobs (All)" CASE 6 thisSectionCode$ = "ppp" thisSectionName$ = "Personals (All)" CASE 7 thisSectionCode$ = "res" thisSectionName$ = "Resumes (All)" CASE 8 thisSectionCode$ = "sss" thisSectionName$ = "For Sale (All)" CASE 9 thisSectionCode$ = "bbb" thisSectionName$ = "Services (All)" END SELECT OutputFile$ = OutputFile$ & """" & thisSectionCode$ & """,""" & thisSectionCode$ & """,""" & thisSectionName$ & """" & $CRLF retryGrab: sectSite$ = httpGrab$("toronto.en.craigslist.ca", "/" & thisSectionCode$ & "/", %TRUE) 'Try and grab the page from the Toronto site IF LEN(sectSite$) < 50 OR INSTR(sectSite$, "