Changes - j0ke.net Open Build Service

@@ -1,664 +0,0 @@ -2002-09-08 16:37 ftobin - - * lib/pyzor/client.py: if None is as the digest to send_digest(), - then just don't send it anyplace (don't die!) - -2002-09-08 15:24 ftobin - - * THANKS: thanks to Robert Schiele for pointing out that the - discovery cgi scripts were 404 - -2002-09-07 23:33 ftobin - - * NEWS, docs/usage.html, lib/pyzor/server.py, scripts/pyzord: - handle HUP handling, which reopens the logfile also, fix USR1 - handling - -2002-09-07 23:33 ftobin - - * UPGRADING: note that upgrades are for 0.3.0 and later - -2002-09-07 22:15 ftobin - - * lib/pyzor/client.py: fix small typo - -2002-09-07 22:11 ftobin - - * NEWS: indicate which encodings we support - -2002-09-07 21:57 ftobin - - * lib/pyzor/client.py: emit errors if too many args are given to - commands - -2002-09-07 18:45 ftobin - - * unittests.py, t/atomic, t/atomic.not: make some test inputs - rfc822 messages, and let the unittests take a hard-coded StringIO - instead - -2002-09-07 03:28 ftobin - - * lib/pyzor/client.py: cleanups - -2002-09-07 02:12 ftobin - - * lib/pyzor/client.py: rename pre-digest to predigest - -2002-09-07 02:08 ftobin - - * lib/pyzor/client.py: add predigest to usage msg - -2002-09-07 02:06 ftobin - - * NEWS: rewording - -2002-09-06 21:12 ftobin - - * NEWS, docs/usage.html, lib/pyzor/__init__.py, - lib/pyzor/client.py, scripts/test.in.0, scripts/test.in.mbox, - scripts/test.sh: implement 'predigest' command - -2002-09-05 20:56 ftobin - - * lib/pyzor/client.py: code cleanup and reorganization, mainly of - digesting morass - -2002-09-04 22:17 ftobin - - * lib/pyzor/: __init__.py, client.py, server.py: rearranging of - iterator classes refactored __iter__ method - -2002-09-04 18:41 ftobin - - * lib/pyzor/client.py: change some instance attrs for some classes - from the name 'file' to 'fp divide up the actual digesting stuff - into smaller stuff so that we can later implement printing out what - is actually digested - -2002-09-04 16:34 ftobin - - * NEWS, unittests.py, docs/usage.html, lib/pyzor/__init__.py, - lib/pyzor/client.py: implement the 'digest' command, along with - moving around some classes for clarity. Renamed the PiecesDigest - stuff to just DataDigest because we don't always break down into - pieces (e.g., if dealing with an atomic message) - -2002-09-04 13:55 ftobin - - * lib/pyzor/: __init__.py, server.py: change some 'while 1' to - 'while True' - -2002-09-04 13:35 ftobin - - * lib/pyzor/__init__.py: add comment about how a mutable class var - is not thread-safe - -2002-09-04 00:07 ftobin - - * MANIFEST, t/atomic, t/atomic.not: add atomic test files - -2002-09-04 00:05 ftobin - - * NEWS, lib/pyzor/__init__.py: bump version - -2002-09-04 00:03 ftobin - - * unittests.py, lib/pyzor/__init__.py: write tests and support for - atomicness - -2002-09-03 23:40 ftobin - - * NEWS: note the atomic message concept - -2002-09-03 23:37 ftobin - - * unittests.py, lib/pyzor/__init__.py: implement 'atomic' message - concept where a message is too small to be divided up - -2002-09-02 23:56 ftobin - - * NEWS, lib/pyzor/client.py: insert rfc822BodyCleaner into the - machine - -2002-09-02 23:42 ftobin - - * lib/pyzor/: __init__.py, client.py, server.py: remove needless - copyright notices from the top of each file - -2002-08-22 18:39 ftobin - - * MANIFEST, unittests.py, lib/pyzor/client.py, t/multipart, - t/multipart.expected: implemented rfc822BodyCleaner class (but - haven't integrated it) also added test for it - -2002-08-20 23:18 ftobin - - * lib/pyzor/client.py: rename a procedure from a noun to a verb - -2002-08-20 23:13 ftobin - - * lib/pyzor/client.py: don't restore the SIGALRM handler; it's - useful to keep it set all the time (also changing it each time is a - syscall) - -2002-08-18 22:00 ftobin - - * NEWS: note that normalization removes HTML tags and lines less - than 8 chars - -2002-08-18 21:56 ftobin - - * unittests.py, lib/pyzor/__init__.py: don't hash lines less than a - certain length normalization removes html tags (in any content - type, for simplicity) added tests for normalization - -2002-08-16 16:16 ftobin - - * lib/pyzor/__init__.py: make code self-documenting - -2002-08-16 12:29 ftobin - - * THANKS, lib/pyzor/__init__.py: fix bug where incorrect percentage - offset is given - - Date: Fri, 16 Aug 2002 09:29:22 +0400 (MSD) From: Roman Suzi - <rnd@onego.ru> Cc: pyzor-users@lists.sourceforge.net Subject: bug?! - - I've found that the following code always give wrong offset: - - offset = line_offsets[int(perc_offset/100 * len(line_offsets))] - - - this is due to the fact that perc_offset/100 produces 0. I - propose to change it to: - - offset = line_offsets[int(perc_offset * len(line_offsets)/100)] - - or - - offset = line_offsets[int(perc_offset*len(line_offsets)/100.)] - - Sincerely yours, Roman A.Suzi -- - Petrozavodsk - Karelia - Russia - - mailto:rnd@onego.ru - - -2002-08-16 12:28 ftobin - - * NEWS: note that /etc/pyzor is fallback - -2002-08-13 16:54 ftobin - - * THANKS: thanks to Bobby Rose for /etc/pyzor suggestion as a - fallback - -2002-08-13 16:51 ftobin - - * lib/pyzor/__init__.py, lib/pyzor/client.py, scripts/pyzord: 'fall - back' to /etc/pyzor if no HOME - or another way of thinking of it, - /etc/pyzor is config dir unless HOME or --homedir is specified - -2002-08-09 17:26 ftobin - - * MANIFEST: add unittests.py - -2002-08-09 17:15 ftobin - - * INSTALL: fix typo about chmodding +r on stuff - -2002-08-09 17:14 ftobin - - * README: pointer to the INSTALL doc - -2002-08-09 16:55 ftobin - - * lib/pyzor/__init__.py: bump version - -2002-08-09 16:54 ftobin - - * NEWS: update NEWS concerning how database file stays open - -2002-08-09 16:41 ftobin - - * lib/pyzor/client.py: fix bug where each report or whitelist is - sent to the each server n times, where n is the number of servers - -2002-08-09 16:40 ftobin - - * docs/usage.html: document command output - -2002-07-30 14:00 ftobin - - * scripts/pyzord: chdir to the root dir - -2002-07-27 22:35 ftobin - - * lib/pyzor/client.py: just some refactoring - -2002-07-27 22:13 ftobin - - * lib/pyzor/server.py, scripts/pyzord: open the gdbm once, and keep - it around - -2002-07-12 16:02 ftobin - - * lib/pyzor/server.py: just a little cleanup - -2002-07-04 16:59 ftobin - - * NEWS, lib/pyzor/server.py: include response code in pyzord.log - -2002-07-03 13:49 ftobin - - * scripts/pyzord: redirect stdin and stdout to other filehandles - -2002-07-02 19:01 ftobin - - * INSTALL, MANIFEST, NEWS, UPGRADING: add an UPGRADING file - -2002-07-02 17:37 ftobin - - * INSTALL: a little instruction on how to change perms for bad - installs - -2002-07-02 17:32 ftobin - - * scripts/pyzord: arg, fix typo in what anonymous is allowed to do - -2002-07-02 16:12 ftobin - - * MANIFEST, scripts/.cvsignore, scripts/accounts, - scripts/config.bob, scripts/bob/accounts, scripts/bob/servers: as - files were moved around, update where everything is and such - -2002-07-02 16:09 ftobin - - * docs/usage.html, lib/pyzor/__init__.py, lib/pyzor/client.py, - scripts/access, scripts/pyzord, scripts/test.sh: relative - filesnames in the config are now treated as relative to the - homedir, and use a --homedir option instead of -c - -2002-07-02 14:09 ftobin - - * INSTALL, NEWS, README: change some references of index.html to - usage.html - -2002-07-02 14:01 ftobin - - * docs/usage.html: add an id to the cvsid p - -2002-07-02 13:59 ftobin - - * MANIFEST, setup.py, docs/index.html, docs/usage.html: rename - docs/index.html to docs/usage.html - -2002-07-02 00:19 ftobin - - * NEWS: clarify a bit more on what http digest signing is - -2002-07-01 22:57 ftobin - - * MANIFEST: add a bunch of files that we now use, such as test info - -2002-07-01 22:56 ftobin - - * setup.py: change my email to neverending, and add docs to to list - -2002-07-01 22:55 ftobin - - * docs/index.html: change default for inform-servers to the 0-3-x - stuff - -2002-07-01 22:53 ftobin - - * lib/pyzor/: __init__.py, client.py: inform_url is for 0-3-x - branch specifically - -2002-07-01 22:49 ftobin - - * docs/index.html: fix typo - -2002-07-01 22:38 ftobin - - * INSTALL: cleanup - -2002-07-01 22:34 ftobin - - * NEWS: news is actually for 0.3.0 - -2002-07-01 13:23 ftobin - - * docs/index.html: fix some spelling errors - -2002-06-30 00:52 ftobin - - * docs/index.html: move around cvs tag - -2002-06-30 00:51 ftobin - - * docs/index.html: add comment for CVS tag - -2002-06-30 00:50 ftobin - - * docs/index.html: add docs - -2002-06-29 23:28 ftobin - - * scripts/pyzord: cleanup - -2002-06-29 23:26 ftobin - - * NEWS: add news for 0.4.0 - -2002-06-29 21:20 ftobin - - * lib/pyzor/client.py: if a message is whitelisted anyplace, then - it counts as a zero count everyplace - -2002-06-29 19:14 ftobin - - * lib/pyzor/__init__.py, lib/pyzor/client.py, lib/pyzor/server.py, - scripts/test.sh: implement whitelisting - -2002-06-19 13:41 ftobin - - * lib/pyzor/__init__.py, lib/pyzor/client.py, lib/pyzor/server.py, - scripts/access, scripts/accounts, scripts/passwd, scripts/pyzord, - scripts/test.sh: add 'info' functionality add test config files - that were forgotten - -2002-06-18 12:14 ftobin - - * scripts/config.bob: forgot to commit this - -2002-06-17 13:18 ftobin - - * unittests.py, lib/pyzor/__init__.py, lib/pyzor/client.py, - lib/pyzor/server.py, scripts/config, scripts/pyzord, - scripts/test.sh: implemented shutdown - -2002-06-09 13:35 ftobin - - * lib/pyzor/__init__.py, lib/pyzor/client.py, lib/pyzor/server.py, - scripts/pyzord: bunch of things to get the test script working - - moving docs out of pyzor/__init__.py, need to put them into a - better format - -2002-06-08 19:58 ftobin - - * scripts/: pyzor, pyzord: change the interpreter line, and fix one - namespace issue - -2002-06-07 12:15 ftobin - - * lib/pyzor/: __init__.py, server.py: increase expire time to 3 - months - -2002-06-06 13:31 ftobin - - * lib/pyzor/__init__.py: actually, hard-code the max threadid, - instead of sys.maxint, in case that's too dynamic - -2002-06-06 13:29 ftobin - - * lib/pyzor/__init__.py: comment out some unused classes, and - ThreadId now uses sys.maxint instead of a hard-coded number - -2002-06-05 22:00 ftobin - - * unittests.py, lib/pyzor/__init__.py, lib/pyzor/client.py, - lib/pyzor/server.py, scripts/pyzord: server side: keys, access - control files - - client side: keys, accounts files - - And yummy unittests! - -2002-06-03 12:40 ftobin - - * INSTALL: more information about where/how to get the gdbm stuff - -2002-05-17 16:58 ftobin - - * lib/pyzor/__init__.py, lib/pyzor/client.py, lib/pyzor/server.py, - scripts/pyzord: shove stuff to cvs because my system is getting - flaky - -2002-05-07 23:26 ftobin - - * INSTALL, NEWS, THANKS, lib/pyzor/__init__.py, - lib/pyzor/client.py, lib/pyzor/server.py, scripts/test.sh: major - changes. - -2002-04-29 17:08 ftobin - - * NEWS, THANKS, lib/pyzor/__init__.py, lib/pyzor/client.py, - scripts/test.sh: fixed major bug where the incorrect exit code is - given - -2002-04-22 13:37 ftobin - - * lib/pyzor/__init__.py: workout docs - -2002-04-22 12:39 ftobin - - * lib/pyzor/__init__.py: document logfile - -2002-04-22 11:59 ftobin - - * lib/pyzor/client.py: tweak the output of ping/checks/reports to - use tabs instead of ': ' as a divider - -2002-04-21 20:38 ftobin - - * lib/pyzor/: __init__.py, client.py: if we get an empty file, - don't croak - -2002-04-21 20:17 ftobin - - * NEWS, lib/pyzor/__init__.py, lib/pyzor/client.py, - lib/pyzor/server.py, scripts/pyzord: add lots of documentation, - bump version, and add CleanupAge for server. - -2002-04-21 19:13 ftobin - - * MANIFEST, scripts/.cvsignore: changes to reflect the new files - used - -2002-04-21 19:04 ftobin - - * scripts/test.sh: check for logfile existence - -2002-04-21 18:56 ftobin - - * INSTALL, lib/pyzor/__init__.py, lib/pyzor/client.py, - lib/pyzor/server.py, scripts/config, scripts/pyzord, - scripts/servers, scripts/test.sh: lots of changes, including - logging, multiple servers support, config files, user/auth with - messages - -2002-04-21 18:43 ftobin - - * scripts/.pyzor: remove this old file - -2002-04-20 21:56 ftobin - - * lib/pyzor/__init__.py: Change readyexec filenames, as per - readyexec 0.2.0 - -2002-04-17 00:40 ftobin - - * NEWS: talk about how we talk to multiple servers - -2002-04-16 20:46 ftobin - - * lib/pyzor/__init__.py, lib/pyzor/client.py, scripts/test.sh: able - to contact multiple servers - -2002-04-16 14:32 ftobin - - * NEWS: talk about how we support ReadyExec - -2002-04-16 14:17 ftobin - - * lib/pyzor/__init__.py: docs on how to use with readyexec - -2002-04-16 14:17 ftobin - - * lib/pyzor/server.py: to help xemacs with highlighting, move the - copyright stuff into a comment, instead of docstring - -2002-04-16 14:12 ftobin - - * MANIFEST: add THANKS - -2002-04-16 14:02 ftobin - - * lib/pyzor/client.py, scripts/pyzor: move most code to client.py, - so readyexec can be used - -2002-04-16 14:01 ftobin - - * scripts/test.in.mbox: add forgotten file - -2002-04-16 14:01 ftobin - - * scripts/test.sh: more graceful handling of errors - -2002-04-16 12:52 ftobin - - * INSTALL: where to find the gdbm tar.gz stuff - -2002-04-16 12:44 ftobin - - * lib/pyzor/.cvsignore: cvsignore to match *.pyc - -2002-04-16 12:43 ftobin - - * INSTALL: direction on how to test for gdbm - -2002-04-16 12:43 ftobin - - * scripts/test.in.mbox: forgot to add this earlier - -2002-04-16 11:56 ftobin - - * lib/pyzor/__init__.py: fix typo - - caught by Rick Macdougall <rickm@nougen.com> - -2002-04-16 11:55 ftobin - - * THANKS: add THANKS file. Rick Macdougall is first entry. - -2002-04-15 19:51 ftobin - - * setup.py: wasn't reading the lib dir correctly - -2002-04-15 19:48 ftobin - - * NEWS: note that no changes were made on the server end - -2002-04-15 19:46 ftobin - - * MANIFEST, NEWS, lib/pyzor/__init__.py, scripts/pyzor, - scripts/test.in.0, scripts/test.sh: now report by default expects a - normal rfc822 message. report now requires --mbox for reporting - entire mailboxes. - - bumped version - -2002-04-15 12:43 ftobin - - * .cvsignore, scripts/.cvsignore: seems like cvsignore files are - nice - -2002-04-14 18:23 ftobin - - * lib/pyzor/server.py, scripts/pyzor, scripts/test.sh: make sure - the database exists before querying it - -2002-04-14 18:08 ftobin - - * lib/pyzor/client.py: typo - -2002-04-14 18:02 ftobin - - * lib/pyzor/client.py: fix ip:port split problem - -2002-04-14 17:37 ftobin - - * MANIFEST: add testing files - -2002-04-14 17:36 ftobin - - * NEWS: add the NEWS - -2002-04-14 17:36 ftobin - - * README: add the README - -2002-04-14 17:33 ftobin - - * lib/pyzor/__init__.py: document the USR1 signal - -2002-04-14 17:30 ftobin - - * lib/pyzor/server.py: remove an erroneous debuging call - -2002-04-14 17:24 ftobin - - * lib/pyzor/__init__.py, scripts/pyzor: doc fixes, and exit false - on timeout - -2002-04-14 17:17 ftobin - - * scripts/pyzor: pyzor client has non-seekable input - -2002-04-14 17:10 ftobin - - * MANIFEST: add setup.py - -2002-04-14 17:04 ftobin - - * scripts/: .pyzor, test.in.0, test.sh: add some testing - -2002-04-14 16:48 ftobin - - * INSTALL: note that python gdbm module is required - -2002-04-14 16:44 ftobin - - * INSTALL: add this wayward file - -2002-04-14 16:33 ftobin - - * lib/pyzor/server.py, scripts/pyzord: add forking for server, and - fix some debugging - -2002-04-14 15:52 ftobin - - * lib/pyzor/__init__.py, lib/pyzor/client.py, lib/pyzor/server.py, - scripts/pyzor, scripts/pyzord: lots of documentation - -2002-04-14 15:19 ftobin - - * lib/pyzor/client.py, lib/pyzor/server.py, scripts/pyzor, - scripts/pyzord: work on args-handling - -2002-04-14 10:19 ftobin - - * MANIFEST, setup.py, lib/pyzor/__init__.py, lib/pyzor/client.py, - lib/pyzor/server.py: add header information and some distribution - files - -2002-04-14 10:00 ftobin - - * COPYING, lib/pyzor/__init__.py, lib/pyzor/client.py, - lib/pyzor/server.py, scripts/pyzor, scripts/pyzord: Initial - revision - -2002-04-14 10:00 ftobin - - * COPYING, lib/pyzor/__init__.py, lib/pyzor/client.py, - lib/pyzor/server.py, scripts/pyzor, scripts/pyzord: import -

@@ -6,6 +6,8 @@ README THANKS UPGRADING +docs/pyzor.1 +docs/pyzord.1 docs/usage.html lib/pyzor/__init__.py lib/pyzor/client.py

@@ -1,3 +1,60 @@ +Noteworthy changes in 0.5.0 +----------------------------------------------------------------- + +Note that the majority of changes in this release were contributed back +from the Debian pyzor package. + + * Man pages for pyzor and pyzord. + + * Changing back to signals for database locking, + rather than threads. It is likely that signals + will be removed again in the future, but the + existing threading changes caused problems. + + * Basic checks on the results of "discover". + + * Extended mbox support throughout the library. + + * Better handling on unknown encodings. + + * Added a --log option to log to a file. + + * Better handling of command-line options. + + * Improved error handling. + +Noteworthy changes in 0.4.x +----------------------------------------------------------------- + + * pyzor client now more gracefully handles base64 and + multipart decoding errors, so that it can be used + over an mbox. + + * pyzor client has new config file option in the [client] + section, Timeout, which specifies a timeout in seconds + for queries to come back to the client. + + * pyzord no longer daemonizes itself, and now writes + it logging to standard output. + + * The following server config options no longer have effect: + PidFile, LogFile. + + * Upped the allowed signed timestamp difference to be up to + 5 minutes (up from 3 minutes). + + * Removed the 'shutdown' command; implementation of + 'meta' commands need to be re-thought. + + * Rewrite of threads locking to access the database. + + * pyzord no longer handles USR1 signals; instead, it now + automatically reorganizes and cleans-up the database daily. + + * Client code now uses threading to catch timeouts, + rather than an alarm signal. + + Noteworthy changes in 0.4.0 -----------------------------------------------------------------

@@ -1,6 +1,6 @@ Metadata-Version: 1.0 Name: pyzor -Version: 0.4.0 +Version: 0.5.0 Summary: networked spam-signature detection Home-page: http://pyzor.sourceforge.net/ Author: Frank J. Tobin

@@ -4,8 +4,9 @@ it complete and free of errors. Frank Tobin ftobin@neverending.org -Rick Macdougall rickm@nougen.com +Rick Macdougall rickm@nougen.com Colin Smith colin@archeus.plus.com Bobby Rose brose@med.wayne.edu Roman Suzi rnd@onego.ru -Robert Schiele schiele@users.sourceforge.net +Robert Schiele schiele@users.sourceforge.net +Tobias Klauser tux_edo@users.sourceforge.net

@@ -0,0 +1,128 @@ +.TH PYZOR 1 "10 Oct 2002" +.SH NAME +pyzor \- spam\-catcher using a collaborative filtering network +.SH SYNOPSIS +\fBpyzor\fP [\fB\-d\fP] [\fB\-\-homedir\fP \fIdir\fP] \fIcommand\fP [\fIcommand\_options\fP] +.SH OPTIONS +.TP +\fB\-d\fP +turn on debugging +.TP +\fB\-\-homedir\fI dir\fP +use dir as the home directory for Pyzor instead of the default ~/.pyzor. +See the files section for more information on what files are inside of the +homedir. +.SH COMMANDS +.TP +\fBcheck\fP[\fB\-\-mbox\fP] +Reads on standard input an RFC 822 (email) message. Exit code is zero (0) +if and only if a match is found and the global whitelist count is zero. +.BR +If \-\-mbox is provided, then the input is assumed to be a unix mailbox, and +all messages in it will be checked. +.BR +If multiple servers are listed in the configuration file, the exit code will +be zero (0) if and only if there is a match found on at least one server +(without it being whitelisted anyplace). +.TP +\fBreport \fP[\fB\-\-mbox\fP] +Reads on standard input an RFC 822 (email) message. Reports to the server a +digest of each message in the mailbox as spam. Writes to standard output a +tuple of (error\-code, message) from the server. +.BR +If \-\-mbox is provided, then the input is assumed to be a unix mailbox, and +all messages in it will be sent to the server. +.TP +\fBwhitelist \fP[\fB\-\-mbox\fP] +Reads on standard input an RFC 822 (email) message. Sends to the server a +digest of each message in the mailbox for whitelisting. Writes to standard +output a tuple of (error\-code, message) from the server. +.BR +If \-\-mbox is provided, then the input is assumed to be a unix mailbox, and +all messages in it will be sent to the server. +.TP +\fBdiscover\fP +Finds Pyzor servers, and writes them to ~/.pyzor/servers. This may +accomplished through querying already-known servers or an HTTP call to a +hard-coded address. +.TP +\fBping\fP +Merely requests a response from the servers. +.TP +\fBgenkey\fP +Based upon a secret passphrase gathered from the user and salt gathered from +/dev/random, prints to standard output a tuple of "salt,key". +Used to put account information into the accounts file. +See the section Using Accounts for more information. +.TP +\fBdigest \fP[\fB\-\-mbox\fP] +Reads on standard input an RFC 822 (email) message. Writes the +digest of the message to standard output. +.BR +If \-\-mbox is provided, then the input is assumed to be a unix +mailbox, each message's digest is written to standard output, separated +by newlines. +.TP +\fBpredigest\fP +Reads on standard input an RFC 822 (email) message. Writes to +standard output the normalized lines of data that are digested, with the +exception that the lines printed have newlines (all whitespace is +removed before digesting). + +.SH USING PYZOR WITH READYEXEC +\fBReadyExec\fP is a system to eliminate the high startup-cost of executing +scripts repeatedly. If you execute pyzor a lot, you might be interested +in installing ReadyExec and using it with pyzor. + +To use pyzor with ReadyExec, the readyexecd.py server needs to be +started as: + + readyexecd.py socket_file pyzor + +socket_file can be any (non\-existing) filename you wish ReadyExec to +use, such as /tmp/pyzor: + + readyexecd.py /tmp/pyzor pyzor + +Individual clients are then executed as: + + readyexec socket_file options command cmd_options + +For example: + + readyexec /tmp/pyzor check + readyexec /tmp/pyzor report + readyexec /tmp/pyzor whitelist \-\-mbox + readyexec /tmp/pyzor \-d ping + +ReadyExec can be found at: http://readyexec.sourceforge.net/ +.SH INTEGRATION WITH MUTT +Add the following line to mutt.conf: + + macro index S "|/usr/bin/pyzor report" + +Then press S on the spam message in mutt to report it with pyzor. +.SH FILES +\fI~/.pyzor/config\fP + +The format of this file is INI-style (name=value, divided into [sections]). +Names are case insensitive. All values which are filenames can have +shell\-style tildes (~) in them. All values which are relative filenames are +interpreted to be relative to the Pyzor homedir. + +\fBDefaults\fP + [client] + ServersFile = servers + AccountsFile = accounts + DiscoverServersURL = http://pyzor.sourceforge.net/cgi-bin/inform\-servers\-0\-3\-x + Timeout = 5 +.SH SEE ALSO +pyzord(1) +.SH AUTHOR +This manpage was originally written by Bastian Kleineidam +<calvin@debian.org> for the Debian distribution of pyzor but may be used +by others. +.BR +The main author of pyzor is Frank J. Tobin <ftobin@neverending.org>. The +main project page for pyzor can be found at +http://sourceforge.net/projects/pyzor

@@ -0,0 +1,44 @@ +.TH PYZORD 1 "10 Oct 2002" +.SH NAME +pyzord \- spam\-catching server +.SH SYNOPSIS +\fBpyzord\fP [\fB\-d\fP] [\fB\-\-homedir\fP \fIdir\fP] + +Note: pyzord does not daemonize itself. + +Note: logging information is written to standard output. + +.SH OPTIONS +.TP +\fB\-d\fP +Turn on debugging +.TP +\fB\-\-homedir\fP \fIdir\fP +use dir as the home directory for Pyzor instead of the default ~/.pyzor. +See the files section for more information on what files are inside of the +homedir. +.SH FILES +\fI~/.pyzor/config\fP + +The format of this file is INI-style (name=value, divided into [sections]). +Names are case insensitive. All values which are filenames can have +shell\-style tildes (~) in them. All values which are relative filenames are +interpreted to be relative to the Pyzor homedir. + +\fBDefaults\fP + [server] + Port = 24441 + ListenAddress = 0.0.0.0 + DigestDB = pyzord.db + PasswdFile = pyzord.passwd + AccessFile = pyzord.access +.SH SEE ALSO +pyzor(1) +.SH AUTHOR +This manpage was originally written by Bastian Kleineidam +<calvin@debian.org> for the Debian distribution of pyzor but may be used +by others. +.BR +The main author of pyzor is Frank J. Tobin <ftobin@neverending.org>. The +main project page for pyzor can be found at +http://sourceforge.net/projects/pyzor

@@ -9,8 +9,8 @@ <body> <h1>Pyzor Usage Documentation</h1> - <p id="cvsid">$Id: usage.html,v 1.7 2002/09/08 03:33:44 ftobin Exp $</p> - + <p id="cvsid">$Id: usage.html,v 1.11 2002-10-10 21:37:08 ftobin Exp $</p> + <ul> <li><a href="#client"><kbd>pyzor</kbd> (client)</a></li> <li><a href="#server"><kbd>pyzord</kbd> (server)</a></li> @@ -21,17 +21,17 @@ </ul> <h2 id="client"><kbd>pyzor</kbd> (client)</h2> - + <pre> pyzor [-d] [--homedir <var class="filename">dir</var>] <var>command</var> [<var>command_options</var>] </pre> - + <h3 id="options">options</h3> - + <dl> <dt><kbd>-d</kbd></dt> <dd>turn on debugging</dd> - + <dt id="pyzor-homedir"><kbd>--homedir <var class="filename">dir</var></kbd></dt> <dd> use <var>dir</var> as the home directory @@ -41,13 +41,13 @@ more information on what files are inside of the homedir. </dd> - + </dl> - + <h3 id="commands">commands</h3> <h4>General Output</h4> - + <p> In general, the output from <kbd>pyzor <var>command</var></kbd> is of the form: @@ -73,13 +73,13 @@ is a match found on at least one server (without it being whitelisted anyplace). </p> - + <p> The command-specfic output for a <kbd>check</kbd> is <samp><var>report-count</var> <var>whitelist-count</var></samp>. </p> </dd> - + <dt id="report"><kbd>report [--mbox]</kbd></dt> <dd> <p> @@ -94,7 +94,7 @@ will be sent to the server. </p> </dd> - + <dt id="whitelist"><kbd>whitelist [--mbox]</kbd></dt> <dd> <p> @@ -109,7 +109,7 @@ will be sent to the server. </p> </dd> - + <dt id="discover"><kbd>discover</kbd></dt> <dd> <p> @@ -119,19 +119,11 @@ servers or an HTTP call to a hard-coded address. </p> </dd> - + <dt id="ping"><kbd>ping</kbd></dt> <dd> <p>Merely requests a response from the servers.</p> </dd> - - <dt id="shutdown"><kbd>shutdown <var>server ...</var></kbd></dt> - <dd> - <p> - Requests each <kbd><var>server</var></kbd> to be shutdown. - Format for <kbd><var>server</var></kbd> is <kbd>host:port</kbd>. - </p> - </dd> <dt id="genkey"><kbd>genkey</kbd></dt> <dd> @@ -143,7 +135,7 @@ See the section <a href="#using-accounts">Using Accounts</a> for more information. </dd> - + <dt id="digest"><kbd>digest [--mbox]</kbd></dt> <dd> <p> @@ -169,10 +161,10 @@ </dd> </dl> - - + + <h3>Using the Pyzor client with procmail</h3> - + <p> To use Pyzor in a procmail system, consider using the following simple recipe. @@ -190,7 +182,7 @@ marked with Pyzor, instead of immediately filtering them into a separate folder: </p> - + <pre> :0 Wc | pyzor check @@ -200,7 +192,7 @@ <h2>Using the Pyzor client with <a href="http://readyexec.sourceforge.net/">ReadyExec</a></h2> - + <p> <a href="http://readyexec.sourceforge.net/">ReadyExec</a> is a system to eliminate the high startup-cost @@ -208,12 +200,12 @@ <kbd>pyzor</kbd> a lot, you might be interested in installing ReadyExec and using it with <kbd>pyzor</kbd>. </p> - + <p> To use pyzor with ReadyExec, the readyexecd.py server needs to be started as: </p> - + <pre> readyexecd.py <var>socket_file</var> pyzor.client.run </pre> @@ -227,19 +219,19 @@ <pre> readyexecd.py /tmp/pyzor pyzor.client.run </pre> - + <p> Individual clients are then executed as: </p> - + <pre> readyexec <var>socket_file</var> <var>options</var> <var>command</var> <var>cmd_options</var> </pre> - + <p> For example: </p> - + <pre> readyexec /tmp/pyzor check readyexec /tmp/pyzor report @@ -249,17 +241,25 @@ <h2 id="server"><kbd>pyzord</kbd> (server)</h2> - + <pre> pyzord [-d] [--homedir <var>dir</var>] </pre> - + + <p> + Note: pyzord does not daemonize itself. + </p> + + <p> + Note: logging information is written to standard output. + </p> + <h3>options</h3> - + <dl> <dt><kbd>-d</kbd></dt> - <dd>Turn on debugging</dd> - + <dd>Turn on debugging (writes information to standard error)</dd> + <dt id="pyzord-homedir"><kbd>--homedir <var class="filename">dir</var></kbd></dt> <dd> use <var>dir</var> as the home directory @@ -269,27 +269,13 @@ more information on what files are inside of the homedir. </dd> - - </dl> - - <h3>Signal handling</h3> - <dl> - <dt>HUP</dt> - <dd> - The server will re-open the logfile. - </dd> - <dt>USR1</dt> - <dd> - The database is 'cleaned up'. - </dd> </dl> - <h2 id="files">Files</h2> - + <h3 id="config"><kbd>~/pyzor/config</kbd></h3> - + <p> The format of this file is INI-style (<kbd><var>name</var>=<var>value</var></kbd>, @@ -300,27 +286,26 @@ filenames are interpreted to be relative to the Pyzor <a href="#pyzor-homedir">homedir</a>. </p> - + <h4>Defaults</h4> - + <pre> <a href="#section-client">[client]</a> <a href="#ServersFile">ServersFile</a> = servers <a href="#AccountsFile">AccountsFile</a> = accounts <a href="#DiscoverServersURL">DiscoverServersURL</a> = http://pyzor.sourceforge.net/cgi-bin/inform-servers-0-3-x +<a href="#Timeout">Timeout</a> = 5 <a href="#section-server">[server]</a> <a href="#Port">Port</a> = 24441 <a href="#ListenAddress">ListenAddress</a> = 0.0.0.0 -<a href="#LogFile">LogFile</a> = pyzord.log -<a href="#PidFile">PidFile</a> = pyzord.pid <a href="#DigestDB">DigestDB</a> = pyzord.db <a href="#PasswdFile">PasswdFile</a> = pyzord.passwd <a href="#AccessFile">AccessFile</a> = pyzord.access </pre> - + <h4>Definitions</h4> - + <h5 id="section-client"><kbd>[client]</kbd> section</h5> <dl> @@ -331,24 +316,24 @@ <kbd>host:port</kbd>. Can be populated with <kbd>pyzor discover</kbd>. </dd> - + <dt id="AccountsFile">AccountsFile</dt> <dd> <p> File containing information about accounts on servers. Format is line-oriented, with each line being: </p> - + <pre> <var>host</var> : <var>port</var> : <var>username</var> : <var>salt,key</var> </pre> - + <p>Example:</p> - + <pre> 127.0.0.1 : 9999 : bob : 227bfb58efaba7c582d9dcb66ab2063d38df2923,8da9f54058c34e383e997f45d6eb74837139f83b </pre> - + <p> See the section <a href="#using-accounts">Using Accounts</a> for more information. @@ -360,45 +345,44 @@ During <kbd>pyzor discover</kbd>, the URL to use in finding servers. </dd> - + + <dt id="Timeout">Timeout</dt> + <dd> + Number of seconds to wait for response to a query. + </dd> + </dl> - + <h5 id="section-server"><kbd>[server]</kbd> section</h5> <dl> <dt id="Port">Port</dt> <dd>port to listen on </dd> - + <dt id="ListenAddress">ListenAddress</dt> <dd>address to listen on</dd> - - <dt id="PidFile">PidFile</dt> - <dd>file to contain the process id of the server</dd> - - <dt id="LogFile">LogFile</dt> - <dd>file to contain server logs</dd> - + <dt id="DigestDB">DigestDB</dt> <dd>file containing the database of digests</dd> - + <dt id="PasswdFile">PasswdFile</dt> <dd> <p> File containing a list of user account information. Line format: </p> - + <pre> <var>username</var> : <var>key</var> </pre> - + <p>Example:</p> <pre> bob : 8da9f54058c34e383e997f45d6eb74837139f83b </pre> - + </dd> - + <dt id="AccessFile">AccessFile</dt> <dd> <p> @@ -409,7 +393,7 @@ <pre> <var>privilege ...</var> : <var>username ...</var> : allow|deny </pre> - + <dl> <dt><var>privilege ...</var></dt> <dd> @@ -419,62 +403,62 @@ The keyword <kbd>all</kbd> can be used to to refer to all commands. </dd> - + <dt><var>username ...</var></dt> <dd> a list of whitespace-separated usernames. - The keyword <kbd>all</kbd> can be used to refer to + The keyword <kbd>all</kbd> can be used to refer to all users. The anonymous user is refereed to as <kbd>anonymous</kbd>. </dd> - + <dt>allow|deny</dt> <dd> Whether or not the specified user(s) can perform the specified privilege(s) on the line. </dd> - + </dl> - + <p> The file is processed from top to bottom, with the first match for user/privilege being the value taken. Every file has the following implicit final rule: </p> - + <pre> all : all : deny </pre> - + <p> If this file is nonexistant, the following default is used: </p> - + <pre> check report ping info : anonymous : allow </pre> - + </dd> - + </dl> - + <h2 id="using-accounts">Using Accounts</h2> - + <p> To get an account on a server requires coordination between the client user and server admin. Use the following steps: </p> - + <ol> <li> User and admin should agree on a username for the user. - Allowed characters for a username are alpha-numerics, + Allowed characters for a username are alpha-numerics, the underscore, and dashes. The normative regular expression it must match is <kbd>^[-\.\w]+$</kbd>. Let us assume they have agreed on 'bob'. </li> - + <li> <p> User generates a key with <kbd>pyzor genkey</kbd>, and @@ -490,13 +474,13 @@ 127.0.0.1 : 9999 : bob : 227bfb58efaba7c582d9dcb66ab2063d38df2923,8da9f54058c34e383e997f45d6eb74837139f83b </pre> </li> - + <li> The user then sends the key (<em>the part to the <strong>right</strong>-hand side of the comma</em>) to the admin. </li> - + <li> <p> The admin adds the following to their @@ -514,29 +498,29 @@ the admin then adds the appropriate permissions to <span class="filename">~/.pyzor/pyzord.access</span>: </p> - + <pre> check report ping info whitelist : bob : allow </pre> - + </li> - + <li> The server needs to be restarted in order for this to take effect. </li> - + </ol> - + <h2 id="razor-diff">User-level Differences from Razor</h2> - + <p> This is a small list of user-recognizable differences between Pyzor and Razor that one might notice if coming from Razor to Pyzor. </p> - + <ul> <li> Pyzor does not consult a whitelist system for you @@ -549,9 +533,9 @@ </ul> <h2 id="copyright">Copyright</h2> - + <p> - Copyright © 2002 Frank J. Tobin + Copyright © 2002-9 Frank J. Tobin <a href="mailto:ftobin@neverending.org">ftobin@neverending.org</a> </p> @@ -576,6 +560,6 @@ along with this program; if not, visit the following URL: <a href="http://www.gnu.org/copyleft/gpl.html">http://www.gnu.org/copyleft/gpl.html</a>. </p> - + </body> </html>

@@ -1,8 +1,8 @@ """networked spam-signature detection""" __author__ = "Frank J. Tobin, ftobin@neverending.org" -__version__ = "0.4.0" -__revision__ = "$Id: __init__.py,v 1.41 2002/09/07 01:12:12 ftobin Exp $" +__version__ = "0.5.0" +__revision__ = "$Id: __init__.py,v 1.43 2002-09-17 15:12:58 ftobin Exp $" import os import os.path @@ -75,7 +75,7 @@ class Username(str): user_pattern = re.compile(r'^[-\.\w]+$') - + def __init__(self, s): self.validate() @@ -87,7 +87,7 @@ class Opname(str): op_pattern = re.compile(r'^[-\.\w]+$') - + def __init__(self, s): self.validate() @@ -106,15 +106,15 @@ def data(self, msg): print msg def warn(self, msg): - if not self.quiet: sys.__stderr__.write('%s\n' % msg) + if not self.quiet: sys.stderr.write('%s\n' % msg) def debug(self, msg): - if self.do_debug: sys.__stderr__.write('%s\n' % msg) + if self.do_debug: sys.stderr.write('%s\n' % msg) class DataDigest(str): # hex output doubles digest size - value_size = sha.digest_size * 2 + value_size = sha.digest_size * 2 def __init__(self, value): if len(value) != self.value_size: @@ -135,7 +135,7 @@ raise ValueError, "offset percentage out of bounds" if not length > 0: raise ValueError, "piece lengths must be positive" - + validate_tuple = staticmethod(validate_tuple) def netstring(self): @@ -155,7 +155,7 @@ self.validate_tuple(perc_offset, length) new_spec.append((perc_offset, length)) - + return new_spec from_netstring = classmethod(from_netstring) @@ -166,7 +166,7 @@ def __init__(self, fp=None): if fp is None: fp = cStringIO.StringIO() - + super(Message, self).__init__(fp) self.setup() @@ -207,13 +207,13 @@ self.setdefault('PV', str(proto_version)) super(ThreadedMessage, self).init_for_sending() - + def ensure_complete(self): if not (self.has_key('PV') and self.has_key('Thread')): raise IncompleteMessageError, \ "doesn't have fields for a ThreadedMessage" super(ThreadedMessage, self).ensure_complete() - + def get_protocol_version(self): return float(self['PV']) @@ -227,8 +227,8 @@ class MacEnvelope(Message): - ts_diff_max = 180 - + ts_diff_max = 300 + def ensure_complete(self): if not (self.has_key('User') and self.has_key('Time') @@ -240,15 +240,15 @@ def get_submsg(self, factory=ThreadedMessage): self.rewindbody() return apply(factory, (self.fp,)) - + def verify_sig(self, user_key): typecheck(user_key, long) - + user = Username(self['User']) ts = int(self['Time']) said_sig = self['Sig'] hashed_user_key = self.hash_key(user_key, user) - + if abs(time.time() - ts) > self.ts_diff_max: raise SignatureError, "timestamp not within allowed range" @@ -261,7 +261,7 @@ def wrap(self, user, key, msg): """This should be used to create a MacEnvelope""" - + typecheck(user, str) typecheck(msg, Message) typecheck(key, long) @@ -294,9 +294,9 @@ """returns lower(H(U + ':' + lower(hex(K))))""" typecheck(key, long) typecheck(user, Username) - + return sha.new("%s:%x" % (Username, key)).hexdigest().lower() - + hash_key = staticmethod(hash_key) @@ -307,9 +307,9 @@ M is message T is decimal epoch timestamp K is hashed_key - + returns a digest object""" - + typecheck(ts, int) typecheck(msg, Message) typecheck(hashed_key, str) @@ -317,7 +317,7 @@ h_msg = self.hash_msg(msg) return sha.new("%s:%d:%s" % (h_msg.digest(), ts, hashed_key)).hexdigest().lower() - + sign_msg = classmethod(sign_msg) @@ -349,7 +349,7 @@ """this is class that should be used to read in Requests of any type. subclasses are responsible for setting 'Op' if they are generating a message""" - + def get_op(self): return self['Op'] @@ -365,7 +365,7 @@ def setup(self): super(Request, self).setup() self.setdefault('Op', self.op) - + class PingRequest(ClientSideRequest): @@ -412,7 +412,7 @@ def __init__(self, code, s): typecheck(code, int) typecheck(s, str) - + super(ErrorResponse, self).__init__() self.setdefault('Code', str(code)) self.setdefault('Diag', s) @@ -424,7 +424,7 @@ full_range = (0, 2**16) ok_range = (1024, full_range[1]) error_value = 0 - + def __init__(self, i): super(ThreadId, self).__init__(i) if not (self.full_range[0] <= self < self.full_range[1]): @@ -448,7 +448,7 @@ typecheck(self[1], int) if len(self) != 2: raise ValueError, "invalid address: %s" % str(self) - + def __str__(self): return (self[0] + ':' + str(self[1])) @@ -468,7 +468,7 @@ assert isinstance(homedir, str) self.homedir = homedir super(Config, self).__init__() - + def get_filename(self, section, option): fn = os.path.expanduser(self.get(section, option)) if not os.path.isabs(fn): @@ -478,14 +478,14 @@ def get_homedir(specified): homedir = os.path.join('/etc', 'pyzor') - + if specified is not None: homedir = specified else: userhome = os.getenv('HOME') if userhome is not None: homedir = os.path.join(userhome, '.pyzor') - + return homedir @@ -501,15 +501,16 @@ dict.""" if kwargs is None: kwargs = {} - + saved = {} for (k, v) in repl.items(): saved[k] = globs[k] globs[k] = v - r = apply(obj, varargs, kwargs) - - globs.update(saved) + try: + r = apply(obj, varargs, kwargs) + finally: + globs.update(saved) return r

@@ -1,5 +1,6 @@ """networked spam-signature detection client""" +import re import os import os.path import socket @@ -8,6 +9,7 @@ import getopt import tempfile import mimetools +import multifile import sha import pyzor @@ -15,20 +17,18 @@ __author__ = pyzor.__author__ __version__ = pyzor.__version__ -__revision__ = "$Id: client.py,v 1.41 2002/09/08 20:37:15 ftobin Exp $" +__revision__ = "$Id: client.py,v 1.48 2003-02-01 10:29:42 ftobin Exp $" randfile = '/dev/random' class Client(object): __slots__ = ['socket', 'output', 'accounts'] - ttl = 4 timeout = 5 max_packet_size = 8192 - + def __init__(self, accounts): signal.signal(signal.SIGALRM, handle_timeout) - self.accounts = accounts self.output = Output() self.build_socket() @@ -42,7 +42,7 @@ msg = InfoRequest(digest) self.send(msg, address) return self.read_response(msg.get_thread()) - + def report(self, digest, spec, address): msg = ReportRequest(digest, spec) self.send(msg, address) @@ -58,18 +58,13 @@ self.send(msg, address) return self.read_response(msg.get_thread()) - def shutdown(self, address): - msg = ShutdownRequest() - self.send(msg, address) - return self.read_response(msg.get_thread()) - def build_socket(self): self.socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) def send(self, msg, address): msg.init_for_sending() account = self.accounts[address] - + mac_msg_str = str(MacEnvelope.wrap(account.username, account.keystuff.key, msg)) @@ -114,25 +109,34 @@ class ServerList(list): inform_url = 'http://pyzor.sourceforge.net/cgi-bin/inform-servers-0-3-x' - + def read(self, serverfile): for line in serverfile: orig_line = line line = line.strip() - if line and not line.startswith('#'): + if line and not line.startswith('#') and \ + re.match('[a-zA-Z0-9.-]+:[0-9]+', line): self.append(pyzor.Address.from_str(line)) class ExecCall(object): __slots__ = ['client', 'servers', 'output'] - + # hard-coded for the moment digest_spec = DataDigestSpec([(20, 3), (60, 3)]) def run(self): debug = 0 - (options, args) = getopt.getopt(sys.argv[1:], 'dh:', ['homedir=']) + log = None + options = None + + try: + (options, args) = getopt.getopt(sys.argv[1:], 'dh:', + ['homedir=', 'log', 'help']) + except getopt.GetoptError: + self.usage() + if len(args) < 1: self.usage() @@ -141,30 +145,37 @@ for (o, v) in options: if o == '-d': debug = 1 - elif o == '-h': + elif o in ('-h', '--help'): self.usage() elif o == '--homedir': specified_homedir = v - - self.output = Output(debug=debug) + elif o == '--log': + log = 1 + self.output = Output(debug=debug) homedir = pyzor.get_homedir(specified_homedir) + if log: + sys.stderr = open(homedir + "/pyzor.log", 'a') + sys.stderr.write("\npyzor[" + repr (os.getpid()) + "]:\n") + config = pyzor.Config(homedir) config.add_section('client') - defaults = {'ServersFile': 'servers', + defaults = {'ServersFile': 'servers', 'DiscoverServersURL': ServerList.inform_url, - 'AccountsFile' : 'accounts', + 'AccountsFile': 'accounts', + 'Timeout': str(Client.timeout), } for k, v in defaults.items(): config.set('client', k, v) - + config.read(os.path.join(homedir, 'config')) - + servers_fn = config.get_filename('client', 'ServersFile') - + Client.timeout = config.getint('client', 'Timeout') + if not os.path.exists(homedir): os.mkdir(homedir) @@ -176,6 +187,12 @@ self.servers = self.get_servers(servers_fn) + if not len(self.servers): + sys.stderr.write("no valid servers found\n") + # Remove the servers file as it only contains invalid entries, + # probably a HTTP error message. + os.remove(servers_fn) + sys.exit(1) self.client = Client(self.get_accounts(config.get_filename('client', 'AccountsFile'))) @@ -197,10 +214,13 @@ def usage(self, s=None): if s is not None: sys.stderr.write("%s\n" % s) - sys.stderr.write("""usage: %s [-d] [--homedir dir] command [cmd_opts] + sys.stderr.write(""" +usage: %s [-d] [--homedir dir] command [cmd_opts] command is one of: check, report, discover, ping, digest, predigest, - genkey, shutdown + genkey + Data is read on standard input (stdin). + """ % sys.argv[0]) sys.exit(2) @@ -208,9 +228,9 @@ def ping(self, args): - getopt.getopt(args[1:], '') - - if len(args) > 1: + try: + getopt.getopt(args[1:], '') + except getopt.GetoptError: self.usage("%s does not take any non-option arguments" % args[0]) runner = ClientRunner(self.client.ping) @@ -219,32 +239,25 @@ runner.run(server, (server,)) return runner.all_ok - - def shutdown(self, args): - (opts, args2) = getopt.getopt(args[1:], '') - if len(args2) > 1: + def info(self, args): + try: + (options, args2) = getopt.getopt(args[1:], '', ['mbox']) + except getopt.GetoptError: self.usage("%s does not take any non-option arguments" % args[0]) - runner = ClientRunner(self.client.shutdown) - - for arg in args2: - server = Address.from_str(arg) - runner.run(server, (server,)) - - return runner.all_ok - + do_mbox = 'msg' - def info(self, args): - getopt.getopt(args[1:], '') - - if len(args) > 1: - self.usage("%s does not take any non-option arguments" % args[0]) + for (o, v) in options: + if o == '--mbox': + do_mbox = 'mbox' runner = InfoClientRunner(self.client.info) - for digest in FileDigester(sys.stdin, self.digest_spec): + for digest in get_input_handler(sys.stdin, self.digest_spec, do_mbox): + if not digest: + continue for server in self.servers: response = runner.run(server, (digest, server)) @@ -252,38 +265,49 @@ def check(self, args): - getopt.getopt(args[1:], '') - - if len(args) > 1: + try: + (options, args2) = getopt.getopt(args[1:], '', ['mbox']) + except getopt.GetoptError: self.usage("%s does not take any non-option arguments" % args[0]) + do_mbox = 'msg' + + for (o, v) in options: + if o == '--mbox': + do_mbox = 'mbox' + runner = CheckClientRunner(self.client.check) - for digest in FileDigester(sys.stdin, self.digest_spec): + for digest in get_input_handler(sys.stdin, self.digest_spec, do_mbox): + if not digest: + continue for server in self.servers: - response = runner.run(server, (digest, server)) - + runner.run(server, (digest, server)) + return (runner.found_hit and not runner.whitelisted) def report(self, args): - (options, args2) = getopt.getopt(args[1:], '', ['mbox']) - do_mbox = False - - if len(args2) > 1: + try: + (options, args2) = getopt.getopt(args[1:], '', ['mbox']) + except getopt.GetoptError: self.usage("%s does not take any non-option arguments" % args[0]) + do_mbox = 'msg' + for (o, v) in options: if o == '--mbox': - do_mbox = True - + do_mbox = "mbox" + all_ok = True - for digest in FileDigester(sys.stdin, self.digest_spec, do_mbox): + for digest in get_input_handler(sys.stdin, self.digest_spec, do_mbox): + if not digest: + continue if not self.send_digest(digest, self.digest_spec, self.client.report): all_ok = False - + return all_ok @@ -297,59 +321,64 @@ for server in self.servers: runner.run(server, (digest, spec, server)) - + return runner.all_ok def whitelist(self, args): - (options, args2) = getopt.getopt(args[1:], '', ['mbox']) - - if len(args2) > 1: + try: + (options, args2) = getopt.getopt(args[1:], '', ['mbox']) + except getopt.GetoptError: self.usage("%s does not take any non-option arguments" % args[0]) - do_mbox = False + do_mbox = "msg" for (o, v) in options: if o == '--mbox': - do_mbox = True - + do_mbox = "mbox" + all_ok = True - for digest in FileDigester(sys.stdin, self.digest_spec, do_mbox): + for digest in get_input_handler(sys.stdin, self.digest_spec, do_mbox): + if not digest: + continue if not self.send_digest(digest, self.digest_spec, self.client.whitelist): all_ok = False - + return all_ok def digest(self, args): - (options, args2) = getopt.getopt(args[1:], '', ['mbox']) - - if len(args2) > 1: + try: + (options, args2) = getopt.getopt(args[1:], '', ['mbox']) + except getopt.GetoptError: self.usage("%s does not take any non-option arguments" % args[0]) - - do_mbox = False + do_mbox = "msg" for (o, v) in options: if o == '--mbox': - do_mbox = True - - for digest in FileDigester(sys.stdin, self.digest_spec, do_mbox): + do_mbox = "mbox" + + for digest in get_input_handler(sys.stdin, self.digest_spec, do_mbox): + if not digest: + continue sys.stdout.write("%s\n" % digest) return True def print_digested(self, args): - getopt.getopt(args[1:], '') - - if len(args) > 1: + try: + getopt.getopt(args[1:], '') + except getopt.GetoptError: self.usage("%s does not take any non-option arguments" % args[0]) + do_mbox = "msg" + def loop(): - for digest in FileDigester(sys.stdin, self.digest_spec): + for digest in get_input_handler(sys.stdin, self.digest_spec, do_mbox): pass modglobal_apply(globals(), {'DataDigester': PrintingDataDigester}, @@ -358,9 +387,9 @@ return True def genkey(self, args): - getopt.getopt(args[1:], '') - - if len(args) > 1: + try: + getopt.getopt(args[1:], '') + except getopt.GetoptError: self.usage("%s does not take any non-option arguments" % args[0]) import getpass @@ -414,7 +443,6 @@ 'report': report, 'ping' : ping, 'genkey': genkey, - 'shutdown': shutdown, 'info': info, 'whitelist': whitelist, 'digest': digest, @@ -428,7 +456,7 @@ """The major workhouse class""" __slots__ = ['_atomic', '_value', '_used_line', '_digest', 'seekable'] - + # minimum line length for it to be included as part # of the digest. I forget the purpose, however. # Someone remind me so I can document it here. @@ -465,8 +493,8 @@ self.seekable = seekable (fp, offsets) = self.get_line_offsets(fp) - - # did we get an empty file? + + # did we get an empty (parsed output) file? if len(offsets) == 0: return @@ -486,9 +514,12 @@ def handle_atomic(self, fp): """we digest everything""" self._atomic = True - fp.seek(0) - for line in fp: - self.handle_line(line) + try: + fp.seek(0) + for line in fp: + self.handle_line(line) + except: + pass def handle_pieced(self, fp, spec, offsets): @@ -519,7 +550,7 @@ cur_offset = fp.tell() newfp = None else: - # we need a seekable file because to make + # we need a seekable file to make # line-based skipping around to be more efficient # than loading the whole thing into memory cur_offset = 0 @@ -527,7 +558,7 @@ offsets = [] - + for line in fp: norm = self.normalize(line) should_handle = self.should_handle_line(norm) @@ -583,9 +614,9 @@ def normalize(self, s): repl = self.unwanted_txt_repl s2 = s + s2 = self.longstr_ptrn.sub(repl, s2) s2 = self.email_ptrn.sub(repl, s2) s2 = self.url_ptrn.sub(repl, s2) - s2 = self.longstr_ptrn.sub(repl, s2) s2 = self.html_tag_ptrn.sub(repl, s2) # make sure we do the whitespace last because some of # the previous patterns rely on whitespace @@ -608,35 +639,41 @@ -class FileDigester(BasicIterator): - __slots__ = ['digester'] +def get_input_handler(fp, spec, style='msg', seekable=False): + """Return an object that can be iterated over + to get all the digests from fp according to spec. + mbox is a boolean""" + if style == 'msg': + return filter(lambda x: x is not None, + (DataDigester(rfc822BodyCleaner(fp), + spec, seekable).get_digest(),) + ) - def __init__(self, fp, spec, mbox=False): - self.digester = iter(get_file_digester(fp, spec, mbox)) - self.output = pyzor.Output() + elif style =='mbox': + return MailboxDigester(fp, spec) - def next(self): - digest = self.digester.next() - self.output.debug("calculated digest: %s" % digest) - return digest + elif style == 'digests': + return JustDigestsIterator(fp) + raise ValueError, "unknown input style" -def get_file_digester(fp, spec, mbox, seekable=False): - """Return an object that can be iterated over - to get all the digests from fp according to spec. - mbox is a boolean""" - if mbox: - return MailboxDigester(fp, spec) +class JustDigestsIterator(BasicIterator): + __slots__ = ['fp'] - return (DataDigester(rfc822BodyCleaner(fp), - spec, seekable).get_digest(),) + def __init__(self, fp): + self.fp = fp + def next(self): + l = fp.readline() + if not l: + raise StopIteration + return l.rstrip() class MailboxDigester(BasicIterator): __slots__ = ['mbox', 'digest_spec', 'seekable'] - + def __init__(self, fp, digest_spec, seekable=False): import mailbox self.mbox = mailbox.PortableUnixMailbox(fp, @@ -645,7 +682,12 @@ self.seekable = seekable def next(self): - next_msg = self.mbox.next() + try: + next_msg = self.mbox.next() + except IOError: + print "Error: Please feed mailbox files in on stdin, i.e." + print " pyzor digest --mbox < my_mbox_file" + next_msg = None if next_msg is None: raise StopIteration return DataDigester(next_msg, self.digest_spec, @@ -655,64 +697,110 @@ class rfc822BodyCleaner(BasicIterator): __slots__ = ['fp', 'multifile', 'curfile', 'type'] - + def __init__(self, fp): msg = mimetools.Message(fp, seekable=0) - self.type = msg.getmaintype() + # Default type is text. See #1529694. + self.type = msg.getmaintype() or "text" self.multifile = None self.curfile = None + # Check if we got a mail or not. Set type to binary if there is no + # 'From:' header and type is text/plain with encoding 7bit. + # 7bit is passed through anyway so nobody cares. + if (not msg.has_key("From") and self.type == 'text' and + msg.subtype == 'plain' and msg.getencoding() == '7bit'): + self.type = 'binary' + + if self.type is '': + self.type = 'text' + if self.type == 'text': encoding = msg.getencoding() - if encoding == '7bit': - self.curfile = msg.fp - else: + self.curfile = msg.fp + if encoding != '7bit': + # fix bad encoding name + if encoding == '8bits': + encoding = '8bit' + + import binascii self.curfile = tempfile.TemporaryFile() - mimetools.decode(msg.fp, self.curfile, encoding) - self.curfile.seek(0) - + try: + mimetools.decode(msg.fp, self.curfile, encoding) + except binascii.Error, e: + sys.stderr.write("%s: %s\n" % (e.__class__, e)) + self.curfile = cStringIO.StringIO() + except ValueError, e: + #sys.stderr.write("%s: %s\n" % (e.__class__, e)) + self.curfile = msg.fp + except multifile.Error, e: + #sys.stderr.write("%s: %s\n" % (e.__class__, e)) + self.curfile = msg.fp + try: + self.curfile.seek(0) + except: + # If we get an error we'll pass the message anyway. + pass + elif self.type == 'multipart': - import multifile - self.multifile = multifile.MultiFile(msg.fp, seekable=False) - self.multifile.push(msg.getparam('boundary')) - self.multifile.next() - self.curfile = self.__class__(self.multifile) + try: + self.multifile = multifile.MultiFile(msg.fp, seekable=False) + self.multifile.push(msg.getparam('boundary')) + self.multifile.next() + self.curfile = self.__class__(self.multifile) + except (TypeError, AttributeError, multifile.Error): + # ignore errors, pass msg as is + self.curfile = msg.fp + self.type = 'binary' if self.type == 'text' or self.type == 'multipart': assert self.curfile is not None + elif self.type == 'binary': + try: + fp.seek(0) + except: + pass + self.curfile = fp else: assert self.curfile is None - + def readline(self): l = '' - if self.type in ('text', 'multipart'): - l = self.curfile.readline() + try: + if self.type in ('text', 'multipart', 'binary'): + l = self.curfile.readline() - if self.type == 'multipart' and not l and self.multifile.next(): - self.curfile = self.__class__(self.multifile) - # recursion. Could get messy if - # we get a bunch of empty multifile parts - l = self.readline() + if self.type == 'multipart' and not l and self.multifile.next(): + self.curfile = self.__class__(self.multifile) + # recursion. Could get messy if + # we get a bunch of empty multifile parts + l = self.readline() + except (TypeError, AttributeError, multifile.Error): + pass return l def next(self): - l = self.readline() - if not l: + try: + l = self.readline() + except multifile.Error, e: + sys.stderr.write("%s: %s\n" % (e.__class__, e)) raise StopIteration - return l + if not l: + raise StopIteration + return l class ClientRunner(object): __slots__ = ['routine', 'all_ok'] - + def __init__(self, routine): self.routine = routine self.setup() - + def setup(self): self.all_ok = True @@ -736,7 +824,7 @@ self.all_ok = False sys.stdout.write(message + str(response.head_tuple()) + '\n') - + class CheckClientRunner(ClientRunner): @@ -750,10 +838,10 @@ self.found_hit = False self.whitelisted = False super(CheckClientRunner, self).setup() - + def handle_response(self, response, message): message += "%s\t" % str(response.head_tuple()) - + if response.is_ok(): wl_count = int(response['WL-Count']) if wl_count > 0: @@ -763,7 +851,7 @@ count = int(response['Count']) if count > 0: self.found_hit = True - + message += "%d\t%d" % (count, wl_count) sys.stdout.write(message + '\n') else: @@ -774,11 +862,11 @@ class InfoClientRunner(ClientRunner): def handle_response(self, response, message): message += "%s\n" % str(response.head_tuple()) - + if response.is_ok(): count = int(response['Count']) message += "\tCount: %d\n" % count - + if count > 0: for f in ('Entered', 'Updated', 'WL-Entered', 'WL-Updated'): if response.has_key(f): @@ -793,9 +881,9 @@ if f is 'WL-Entered': message += ("\tWhiteList Count: %d\n" % int(response['WL-Count'])) - + message += ("\t%s: %s\n" % (f, stringed)) - + sys.stdout.write(message) else: sys.stderr.write(message) @@ -833,13 +921,13 @@ # be removed if self[1] is None: raise ValueError, "no key information" - + for x in self: if not (isinstance(x, long) or x is None): raise ValueError, "Keystuff must be long's or None's" # make sure we didn't get all None's - if not filter(lambda x: x is not None, self): + if not filter(lambda x: x is not None, self): raise ValueError, "keystuff can't be all None's" def from_hexstr(self, s): @@ -869,10 +957,10 @@ class AccountsDict(dict): """Key is pyzor.Address, value is Account When getting, defaults to anonymous_account""" - + anonymous_account = Account((pyzor.anonymous_user, Keystuff((None, 0L)))) - + def __setitem__(self, k, v): typecheck(k, pyzor.Address) typecheck(v, Account) @@ -893,7 +981,7 @@ host : port ; username : keystuff """ __slots__ = ['fp', 'lineno', 'output'] - + def __init__(self, fp): self.fp = fp self.lineno = 0 @@ -906,7 +994,7 @@ while 1: orig_line = self.fp.readline() self.lineno += 1 - + if not orig_line: raise StopIteration line = orig_line.strip() @@ -914,7 +1002,7 @@ continue fields = line.split(':') fields = map(lambda x: x.strip(), fields) - + if len(fields) != 4: self.output.warn("account file: invalid line %d: wrong number of parts" % self.lineno) @@ -939,5 +1027,5 @@ def download(url, outfile): - import urllib - urllib.urlretrieve(url, outfile) + import urllib2 + open(outfile, "wb").write(urllib2.urlopen(url).read())

@@ -4,7 +4,6 @@ import os import sys -import signal import SocketServer import time import gdbm @@ -17,7 +16,7 @@ __author__ = pyzor.__author__ __version__ = pyzor.__version__ -__revision__ = "$Id: server.py,v 1.25 2002/09/08 03:33:44 ftobin Exp $" +__revision__ = "$Id: server.py,v 1.29 2002-10-09 00:45:45 ftobin Exp $" class AuthorizationError(pyzor.CommError): @@ -343,41 +342,77 @@ -class DBHandle(object): - __slots__ = ['output'] - db_lock = threading.Lock() - max_age = 3600*24*30*4 # 3 months - db = None +class DBHandle(Singleton): + __slots__ = ['output', 'initialized'] + db_lock = threading.Lock() + max_age = 3600*24*30*4 # 3 months + db = None + sync_period = 60 + reorganize_period = 3600*24 # 1 day def __init__(self): assert self.db is not None, "database was not initialized" - self.db_lock.acquire() def initialize(self, fn, mode): self.output = Output() self.db = gdbm.open(fn, mode) + self.start_reorganizing() + self.start_syncing() initialize = classmethod(initialize) - def __del__(self): - self.db.sync() - self.db_lock.release() - + def apply_locking_method(self, method, varargs=(), kwargs={}): + # just so we don't carry around a mutable kwargs + if kwargs == {}: + kwargs = {} + self.output.debug("acquiring lock") + self.db_lock.acquire() + self.output.debug("acquired lock") + try: + result = apply(method, varargs, kwargs) + finally: + self.output.debug("releasing lock") + self.db_lock.release() + self.output.debug("released lock") + return result + apply_locking_method = classmethod(apply_locking_method) + def __getitem__(self, key): + return self.apply_locking_method(self._really_getitem, (key,)) + + def _really_getitem(self, key): return self.db[key] def __setitem__(self, key, value): + self.apply_locking_method(self._really_setitem, (key, value)) + + def _really_setitem(self, key, value): self.db[key] = value - def __delitem__(self, key): - del self.db[key] + def start_syncing(self): + self.apply_locking_method(self._really_sync) + self.sync_timer = threading.Timer(self.sync_period, + self.start_syncing) + self.sync_timer.start() + start_syncing = classmethod(start_syncing) - def cleanup(self): - self.output.debug("cleaning up the database") + def _really_sync(self): + self.db.sync() + _really_sync = classmethod(_really_sync) + + def start_reorganizing(self): + self.apply_locking_method(self._really_reorganize) + self.reorganize_timer = threading.Timer(self.reorganize_period, + self.start_reorganizing) + self.reorganize_timer.start() + start_reorganizing = classmethod(start_reorganizing) + + def _really_reorganize(self): + self.output.debug("reorganizing the database") key = self.db.firstkey() breakpoint = time.time() - self.max_age while key is not None: - rec = Record.from_str(self[key]) + rec = Record.from_str(self.db[key]) delkey = None if rec.r_updated < breakpoint: self.output.debug("deleting key %s" % key) @@ -385,14 +420,11 @@ key = self.db.nextkey(key) if delkey: del self.db[delkey] - self.db.reorganize() - + _really_reorganize = classmethod(_really_reorganize) class Server(SocketServer.ThreadingUDPServer, object): - ttl = 4 - timeout = 3 max_packet_size = 8192 time_diff_allowance = 180 @@ -435,8 +467,6 @@ def handle(self): - do_exit = False - try: self._really_handle() except UnsupportedVersionError, e: @@ -447,14 +477,10 @@ # We assume that KeyErrors are due to not # finding a key in the RFC822 message self.handle_error(400, "Bad request: %s" % e) - except TimeoutError, e: - self.handle_error(503, "Gateway timeout: %s" % e) except AuthorizationError, e: self.handle_error(401, "Unauthorized: %s" % e) except SignatureError, e: self.handle_error(401, "Unauthorized, Signature Error: %s" % e) - except SystemExit, e: - do_exit = True except Exception, e: self.handle_error(500, "Internal Server Error: %s" % e) traceback.print_exc() @@ -470,11 +496,6 @@ self.output.debug("sending: %s" % repr(msg_str)) self.wfile.write(msg_str) - if do_exit: - db_hold = DBHandle() # to keep the db consistent - self.finish() - os.kill(self.server.pid, signal.SIGQUIT) - def _really_handle(self): """handle() without the exception handling""" @@ -597,14 +618,9 @@ self.out_msg['WL-Count'] = "%d" % wl_count - def handle_shutdown(self): - raise SystemExit - - dispatches = { 'check': handle_check, 'report': handle_report, 'ping': None, 'info': handle_info, - 'shutdown': handle_shutdown, 'whitelist': handle_whitelist, }

@@ -1,4 +1,8 @@ -#!/usr/bin/python2 +#!/usr/bin/python + +import os +# set umask +os.umask(0077) import pyzor.client pyzor.client.run()

@@ -1,9 +1,8 @@ -#!/usr/bin/python2 +#!/usr/bin/python import os import os.path import sys -import signal import getopt import pyzor import pyzor.server @@ -11,20 +10,15 @@ _author__ = pyzor.__author__ __version__ = pyzor.__version__ -__revision__ = "$Id: pyzord,v 1.20 2002/09/08 03:33:44 ftobin Exp $" +__revision__ = "$Id: pyzord,v 1.23 2002-10-09 00:33:44 ftobin Exp $" +progname = 'pyzord' default_anonymous_allows = map(pyzor.Opname, ['check', 'report', 'ping', 'info']) -def cleanup_server_handler(signum, frame): - pyzor.server.DBHandle().cleanup() - -def reopen_log_handler(signum, frame): - server.replace_log(pyzor.server.Log(open(log_fn, 'a'))) - def usage(): - sys.stderr.write("usage: %s [-d] [--homedir dir]\n" % sys.argv[0]) + sys.stderr.write("usage: %s [-d] [--homedir dir]\n" % progname) sys.exit(1) @@ -33,14 +27,14 @@ if os.path.exists(access_fn): pyzor.server.AccessFile(open(access_fn)).feed_into(server.acl) else: - output.warn("%s does not exist; using default ACL; allowing anonymous to do %s" + output.warn("%s does not exist; using default ACL: allowing anonymous to do %s" % (access_fn, default_anonymous_allows)) for op in default_anonymous_allows: server.acl.add_entry(pyzor.server.ACLEntry((pyzor.anonymous_user, op, True))) - + def load_passwd_file(access_fn, server): server.passwd = pyzor.server.Passwd() if os.path.exists(passwd_fn): @@ -51,6 +45,9 @@ ######################################################################## # functions above, run below +# set umask +os.umask(0077) + debug = 0 (options, args) = getopt.getopt(sys.argv[1:], 'dh:', ['homedir=']) if len(args) != 0: @@ -73,8 +70,6 @@ defaults = {'port': '24441', 'listenaddress': '0.0.0.0', - 'logfile': 'pyzord.log', - 'pidfile': 'pyzord.pid', 'digestdb': 'pyzord.db', 'passwdfile': 'pyzord.passwd', 'accessfile': 'pyzord.access', @@ -92,8 +87,6 @@ port = config.getint('server', 'port') listen_adr = config.get('server', 'ListenAddress') -log_fn = config.get_filename('server', 'logfile') -pid_fn = config.get_filename('server', 'pidfile') dbfile = config.get_filename('server', 'DigestDB') passwd_fn = config.get_filename('server', 'passwdfile') access_fn = config.get_filename('server', 'accessfile') @@ -101,48 +94,12 @@ output = pyzor.Output(debug=debug) -logfile = open(log_fn, 'a') pyzor.server.DBHandle.initialize(dbfile, 'c') - server = pyzor.server.Server((listen_adr, port), - pyzor.server.Log(logfile)) + pyzor.server.Log(sys.stdout)) load_passwd_file(passwd_fn, server) load_access_file(access_fn, server) -if not debug: - # This step is required so that the new process is guaranteed - # not to be a process group leader. The next step, setsid(), - # fails if you're a process group leader. - pid = os.fork() - if pid != 0: - os._exit(0) - - # setsid() to become a process group and session group leader. - # Since a controlling terminal is associated with a session, - # and this new session has not yet acquired a controlling - # terminal our process now has no controlling terminal, - # which is a Good Thing for daemons. - os.setsid() - - pid = os.fork() - if pid != 0: - os._exit(0) - - sys.stdin = sys.__stdin__ = open('/dev/null') - sys.stdout = sys.__stdout__ = open('/dev/null', 'a') - #sys.stderr = sys.__stderr__ = open('/dev/null', 'a') - -if pid_fn is not None: - f = open(pid_fn, 'w') - f.write("%d\n" % os.getpid()) - f.close() - -# we shouldn't chdir because that prevents the re-opening -# of logfiles and the database if they are relative paths -##os.chdir("/") - -signal.signal(signal.SIGUSR1, cleanup_server_handler) -signal.signal(signal.SIGHUP, reopen_log_handler) server.serve_forever()

@@ -3,47 +3,82 @@ # HOME so it finds the right .pyzor export HOME=. export PYTHONPATH=../lib -PYZOR="./pyzor --homedir ." -PYZOR_BOB="./pyzor --homedir bob" -PYZORD="./pyzord --homedir ." -alias check="$PYZOR check < test.in.0" -alias check.bob="$PYZOR check < test.in.0" -kill_server() +start_server() +{ + ./pyzord --homedir . > pyzord.log & + PYZORD_PID=$! + echo "started server with pid $PYZORD_PID" +} + + +pyzor() +{ + ./pyzor --homedir . "$@" +} + + +pyzor_bob() +{ + pyzor --homedir bob "$@" +} + + +check() +{ + pyzor check < test.in.0 +} + + +check_bob() { - echo "killing server" - pidfile='pyzord.pid' - [ -e $pidfile ] && kill `cat $pidfile` + pyzor_bob check < test.in.0 } + +kill_server() +{ + echo "killing server (pid $PYZORD_PID)" + kill $PYZORD_PID +} + fail() { echo "failed: $1" - kill_server exit 1; } setcount() { - count=`$PYZOR check < test.in.0 | cut -f 3` + count=`pyzor check < test.in.0 | cut -f 3` } setcount_bob() { - count=`$PYZOR check < test.in.0 | cut -f 3` + count=`pyzor_bob check < test.in.0 | cut -f 3` } fail_cmp() { - fail "got $1; expected $2" + fail "got $1; expected $2" } rm -f pyzord.* echo "starting server" -$PYZORD || fail +start_server + +# sleep to give it time to start listening +sleep 2 +[ ! -z $PYZORD_PID ] || fail "we didn't get a process id for the server!" +kill -0 $PYZORD_PID || fail "process is dead" + +trap kill_server 0 + +echo "anonymous: pinging" +pyzor ping || fail echo "anonymous: ensuring a count of 0 at start" setcount @@ -54,19 +89,19 @@ echo "bob: ensuring a count of 0 at start" setcount_bob [ ${count:--1} = 0 ] || fail -check.bob && fail +check_bob && fail echo "anonymous: reporting" -$PYZOR report < test.in.0 && fail +pyzor report < test.in.0 && fail echo "anonymous: reporting" -$PYZOR report < test.in.0 && fail +pyzor report < test.in.0 && fail echo "bob: reporting" -$PYZOR_BOB report < test.in.0 || fail +pyzor_bob report < test.in.0 || fail echo "bob: reporting" -$PYZOR_BOB report < test.in.0 || fail +pyzor_bob report < test.in.0 || fail echo "anonymous: counting reports" @@ -78,38 +113,38 @@ echo "bob: counting reports" setcount_bob [ ${count:--1} = 2 ] || fail_cmp ${count:--1} 2 -check.bob || fail "checking failed" +check_bob || fail "checking failed" echo "bob: reporting a mailbox" -$PYZOR_BOB report --mbox < test.in.mbox || fail +pyzor_bob report --mbox < test.in.mbox || fail echo "bob: counting reports" setcount_bob [ ${count:--1} = 3 ] || fail_cmp ${count:--1} 3 -check.bob || fail "checking exit code failed" +check_bob || fail "checking exit code failed" echo "bob: getting info" # check exit -$PYZOR_BOB info < test.in.0 || fail +pyzor_bob info < test.in.0 || fail # check lines -[ `$PYZOR_BOB info < test.in.0 | wc -l` = 7 ] || fail +[ `pyzor_bob info < test.in.0 | wc -l` = 7 ] || fail echo "anonymous: whitelisting" -$PYZOR whitelist < test.in.0 && fail +pyzor whitelist < test.in.0 && fail echo "bob: whitelisting" -$PYZOR_BOB whitelist < test.in.0 || fail +pyzor_bob whitelist < test.in.0 || fail echo "bob: getting info" # check exit -$PYZOR_BOB info < test.in.0 || fail +pyzor_bob info < test.in.0 || fail # check lines -[ `$PYZOR_BOB info < test.in.0 | wc -l` = 7 ] || fail +[ `pyzor_bob info < test.in.0 | wc -l` = 7 ] || fail echo "bob: counting reports" @@ -118,16 +153,7 @@ check && fail -echo "anonymous: pinging" -$PYZOR ping || fail - echo "checking for logfile" [ -s pyzord.log ] || fail -echo "anonymous: shutting down server" -$PYZOR shutdown 127.0.0.1:9999 && fail - -echo "bob: shutting down server" -$PYZOR_BOB shutdown 127.0.0.1:9999 || fail - echo "passed"

@@ -12,12 +12,12 @@ distutils.core.setup( name = 'pyzor', version = pyzor.__version__, description = 'networked spam-signature detection', - long_description = long_description, + long_description = long_description, author = 'Frank J. Tobin', author_email = 'ftobin@neverending.org', - license = 'GPL', - platforms = 'POSIX', - keywords = 'spam', + license = 'GPL', + platforms = 'POSIX', + keywords = 'spam', url = 'http://pyzor.sourceforge.net/', scripts=['scripts/pyzor', 'scripts/pyzord'], package_dir = {'': 'lib'},

@@ -7,7 +7,7 @@ from pyzor.server import * from pyzor.client import * -__revision__ = "$Id: unittests.py,v 1.8 2002/09/07 22:45:03 ftobin Exp $" +__revision__ = "$Id: unittests.py,v 1.8 2002-09-07 22:45:03 ftobin Exp $" class ACLTest(unittest.TestCase):

[-] [+]	Added	pyzor.changes
@@ -0,0 +1,5 @@ +------------------------------------------------------------------- +Tue Aug 30 20:05:22 UTC 2011 - cs@linux-administrator.com + +- update to release 0.5.0 +
[-] [+]	Changed	pyzor.spec ^
@@ -1,8 +1,8 @@ # norootforbuild Name: pyzor -Version: 0.4.0 -Release: 0 +Version: 0.5.0 +Release: 1 Summary: A collaborative, networked system to detect and block spam License: GPL Group: Productivity/Networking/Email/Utilities
[-] [+]	Deleted	pyzor-0.4.0.tar.bz2/ChangeLog ^
@@ -1,664 +0,0 @@ -2002-09-08 16:37 ftobin - - * lib/pyzor/client.py: if None is as the digest to send_digest(), - then just don't send it anyplace (don't die!) - -2002-09-08 15:24 ftobin - - * THANKS: thanks to Robert Schiele for pointing out that the - discovery cgi scripts were 404 - -2002-09-07 23:33 ftobin - - * NEWS, docs/usage.html, lib/pyzor/server.py, scripts/pyzord: - handle HUP handling, which reopens the logfile also, fix USR1 - handling - -2002-09-07 23:33 ftobin - - * UPGRADING: note that upgrades are for 0.3.0 and later - -2002-09-07 22:15 ftobin - - * lib/pyzor/client.py: fix small typo - -2002-09-07 22:11 ftobin - - * NEWS: indicate which encodings we support - -2002-09-07 21:57 ftobin - - * lib/pyzor/client.py: emit errors if too many args are given to - commands - -2002-09-07 18:45 ftobin - - * unittests.py, t/atomic, t/atomic.not: make some test inputs - rfc822 messages, and let the unittests take a hard-coded StringIO - instead - -2002-09-07 03:28 ftobin - - * lib/pyzor/client.py: cleanups - -2002-09-07 02:12 ftobin - - * lib/pyzor/client.py: rename pre-digest to predigest - -2002-09-07 02:08 ftobin - - * lib/pyzor/client.py: add predigest to usage msg - -2002-09-07 02:06 ftobin - - * NEWS: rewording - -2002-09-06 21:12 ftobin - - * NEWS, docs/usage.html, lib/pyzor/__init__.py, - lib/pyzor/client.py, scripts/test.in.0, scripts/test.in.mbox, - scripts/test.sh: implement 'predigest' command - -2002-09-05 20:56 ftobin - - * lib/pyzor/client.py: code cleanup and reorganization, mainly of - digesting morass - -2002-09-04 22:17 ftobin - - * lib/pyzor/: __init__.py, client.py, server.py: rearranging of - iterator classes refactored __iter__ method - -2002-09-04 18:41 ftobin - - * lib/pyzor/client.py: change some instance attrs for some classes - from the name 'file' to 'fp divide up the actual digesting stuff - into smaller stuff so that we can later implement printing out what - is actually digested - -2002-09-04 16:34 ftobin - - * NEWS, unittests.py, docs/usage.html, lib/pyzor/__init__.py, - lib/pyzor/client.py: implement the 'digest' command, along with - moving around some classes for clarity. Renamed the PiecesDigest - stuff to just DataDigest because we don't always break down into - pieces (e.g., if dealing with an atomic message) - -2002-09-04 13:55 ftobin - - * lib/pyzor/: __init__.py, server.py: change some 'while 1' to - 'while True' - -2002-09-04 13:35 ftobin - - * lib/pyzor/__init__.py: add comment about how a mutable class var - is not thread-safe - -2002-09-04 00:07 ftobin - - * MANIFEST, t/atomic, t/atomic.not: add atomic test files - -2002-09-04 00:05 ftobin - - * NEWS, lib/pyzor/__init__.py: bump version - -2002-09-04 00:03 ftobin - - * unittests.py, lib/pyzor/__init__.py: write tests and support for - atomicness - -2002-09-03 23:40 ftobin - - * NEWS: note the atomic message concept - -2002-09-03 23:37 ftobin - - * unittests.py, lib/pyzor/__init__.py: implement 'atomic' message - concept where a message is too small to be divided up - -2002-09-02 23:56 ftobin - - * NEWS, lib/pyzor/client.py: insert rfc822BodyCleaner into the - machine - -2002-09-02 23:42 ftobin - - * lib/pyzor/: __init__.py, client.py, server.py: remove needless - copyright notices from the top of each file - -2002-08-22 18:39 ftobin - - * MANIFEST, unittests.py, lib/pyzor/client.py, t/multipart, - t/multipart.expected: implemented rfc822BodyCleaner class (but - haven't integrated it) also added test for it - -2002-08-20 23:18 ftobin - - * lib/pyzor/client.py: rename a procedure from a noun to a verb - -2002-08-20 23:13 ftobin - - * lib/pyzor/client.py: don't restore the SIGALRM handler; it's - useful to keep it set all the time (also changing it each time is a - syscall) - -2002-08-18 22:00 ftobin - - * NEWS: note that normalization removes HTML tags and lines less - than 8 chars - -2002-08-18 21:56 ftobin - - * unittests.py, lib/pyzor/__init__.py: don't hash lines less than a - certain length normalization removes html tags (in any content - type, for simplicity) added tests for normalization - -2002-08-16 16:16 ftobin - - * lib/pyzor/__init__.py: make code self-documenting - -2002-08-16 12:29 ftobin - - * THANKS, lib/pyzor/__init__.py: fix bug where incorrect percentage - offset is given - - Date: Fri, 16 Aug 2002 09:29:22 +0400 (MSD) From: Roman Suzi - <rnd@onego.ru> Cc: pyzor-users@lists.sourceforge.net Subject: bug?! - - I've found that the following code always give wrong offset: - - offset = line_offsets[int(perc_offset/100 * len(line_offsets))] - - - this is due to the fact that perc_offset/100 produces 0. I - propose to change it to: - - offset = line_offsets[int(perc_offset * len(line_offsets)/100)] - - or - - offset = line_offsets[int(perc_offsetlen(line_offsets)/100.)] - - Sincerely yours, Roman A.Suzi -- - Petrozavodsk - Karelia - Russia - - mailto:rnd@onego.ru - - -2002-08-16 12:28 ftobin - - NEWS: note that /etc/pyzor is fallback - -2002-08-13 16:54 ftobin - - * THANKS: thanks to Bobby Rose for /etc/pyzor suggestion as a - fallback - -2002-08-13 16:51 ftobin - - * lib/pyzor/__init__.py, lib/pyzor/client.py, scripts/pyzord: 'fall - back' to /etc/pyzor if no HOME - or another way of thinking of it, - /etc/pyzor is config dir unless HOME or --homedir is specified - -2002-08-09 17:26 ftobin - - * MANIFEST: add unittests.py - -2002-08-09 17:15 ftobin - - * INSTALL: fix typo about chmodding +r on stuff - -2002-08-09 17:14 ftobin - - * README: pointer to the INSTALL doc - -2002-08-09 16:55 ftobin - - * lib/pyzor/__init__.py: bump version - -2002-08-09 16:54 ftobin - - * NEWS: update NEWS concerning how database file stays open - -2002-08-09 16:41 ftobin - - * lib/pyzor/client.py: fix bug where each report or whitelist is - sent to the each server n times, where n is the number of servers - -2002-08-09 16:40 ftobin - - * docs/usage.html: document command output - -2002-07-30 14:00 ftobin - - * scripts/pyzord: chdir to the root dir - -2002-07-27 22:35 ftobin - - * lib/pyzor/client.py: just some refactoring - -2002-07-27 22:13 ftobin - - * lib/pyzor/server.py, scripts/pyzord: open the gdbm once, and keep - it around - -2002-07-12 16:02 ftobin - - * lib/pyzor/server.py: just a little cleanup - -2002-07-04 16:59 ftobin - - * NEWS, lib/pyzor/server.py: include response code in pyzord.log - -2002-07-03 13:49 ftobin - - * scripts/pyzord: redirect stdin and stdout to other filehandles - -2002-07-02 19:01 ftobin - - * INSTALL, MANIFEST, NEWS, UPGRADING: add an UPGRADING file - -2002-07-02 17:37 ftobin - - * INSTALL: a little instruction on how to change perms for bad - installs - -2002-07-02 17:32 ftobin - - * scripts/pyzord: arg, fix typo in what anonymous is allowed to do - -2002-07-02 16:12 ftobin - - * MANIFEST, scripts/.cvsignore, scripts/accounts, - scripts/config.bob, scripts/bob/accounts, scripts/bob/servers: as - files were moved around, update where everything is and such - -2002-07-02 16:09 ftobin - - * docs/usage.html, lib/pyzor/__init__.py, lib/pyzor/client.py, - scripts/access, scripts/pyzord, scripts/test.sh: relative - filesnames in the config are now treated as relative to the - homedir, and use a --homedir option instead of -c - -2002-07-02 14:09 ftobin - - * INSTALL, NEWS, README: change some references of index.html to - usage.html - -2002-07-02 14:01 ftobin - - * docs/usage.html: add an id to the cvsid p - -2002-07-02 13:59 ftobin - - * MANIFEST, setup.py, docs/index.html, docs/usage.html: rename - docs/index.html to docs/usage.html - -2002-07-02 00:19 ftobin - - * NEWS: clarify a bit more on what http digest signing is - -2002-07-01 22:57 ftobin - - * MANIFEST: add a bunch of files that we now use, such as test info - -2002-07-01 22:56 ftobin - - * setup.py: change my email to neverending, and add docs to to list - -2002-07-01 22:55 ftobin - - * docs/index.html: change default for inform-servers to the 0-3-x - stuff - -2002-07-01 22:53 ftobin - - * lib/pyzor/: __init__.py, client.py: inform_url is for 0-3-x - branch specifically - -2002-07-01 22:49 ftobin - - * docs/index.html: fix typo - -2002-07-01 22:38 ftobin - - * INSTALL: cleanup - -2002-07-01 22:34 ftobin - - * NEWS: news is actually for 0.3.0 - -2002-07-01 13:23 ftobin - - * docs/index.html: fix some spelling errors - -2002-06-30 00:52 ftobin - - * docs/index.html: move around cvs tag - -2002-06-30 00:51 ftobin - - * docs/index.html: add comment for CVS tag - -2002-06-30 00:50 ftobin - - * docs/index.html: add docs - -2002-06-29 23:28 ftobin - - * scripts/pyzord: cleanup - -2002-06-29 23:26 ftobin - - * NEWS: add news for 0.4.0 - -2002-06-29 21:20 ftobin - - * lib/pyzor/client.py: if a message is whitelisted anyplace, then - it counts as a zero count everyplace - -2002-06-29 19:14 ftobin - - * lib/pyzor/__init__.py, lib/pyzor/client.py, lib/pyzor/server.py, - scripts/test.sh: implement whitelisting - -2002-06-19 13:41 ftobin - - * lib/pyzor/__init__.py, lib/pyzor/client.py, lib/pyzor/server.py, - scripts/access, scripts/accounts, scripts/passwd, scripts/pyzord, - scripts/test.sh: add 'info' functionality add test config files - that were forgotten - -2002-06-18 12:14 ftobin - - * scripts/config.bob: forgot to commit this - -2002-06-17 13:18 ftobin - - * unittests.py, lib/pyzor/__init__.py, lib/pyzor/client.py, - lib/pyzor/server.py, scripts/config, scripts/pyzord, - scripts/test.sh: implemented shutdown - -2002-06-09 13:35 ftobin - - * lib/pyzor/__init__.py, lib/pyzor/client.py, lib/pyzor/server.py, - scripts/pyzord: bunch of things to get the test script working - - moving docs out of pyzor/__init__.py, need to put them into a - better format - -2002-06-08 19:58 ftobin - - * scripts/: pyzor, pyzord: change the interpreter line, and fix one - namespace issue - -2002-06-07 12:15 ftobin - - * lib/pyzor/: __init__.py, server.py: increase expire time to 3 - months - -2002-06-06 13:31 ftobin - - * lib/pyzor/__init__.py: actually, hard-code the max threadid, - instead of sys.maxint, in case that's too dynamic - -2002-06-06 13:29 ftobin - - * lib/pyzor/__init__.py: comment out some unused classes, and - ThreadId now uses sys.maxint instead of a hard-coded number - -2002-06-05 22:00 ftobin - - * unittests.py, lib/pyzor/__init__.py, lib/pyzor/client.py, - lib/pyzor/server.py, scripts/pyzord: server side: keys, access - control files - - client side: keys, accounts files - - And yummy unittests! - -2002-06-03 12:40 ftobin - - * INSTALL: more information about where/how to get the gdbm stuff - -2002-05-17 16:58 ftobin - - * lib/pyzor/__init__.py, lib/pyzor/client.py, lib/pyzor/server.py, - scripts/pyzord: shove stuff to cvs because my system is getting - flaky - -2002-05-07 23:26 ftobin - - * INSTALL, NEWS, THANKS, lib/pyzor/__init__.py, - lib/pyzor/client.py, lib/pyzor/server.py, scripts/test.sh: major - changes. - -2002-04-29 17:08 ftobin - - * NEWS, THANKS, lib/pyzor/__init__.py, lib/pyzor/client.py, - scripts/test.sh: fixed major bug where the incorrect exit code is - given - -2002-04-22 13:37 ftobin - - * lib/pyzor/__init__.py: workout docs - -2002-04-22 12:39 ftobin - - * lib/pyzor/__init__.py: document logfile - -2002-04-22 11:59 ftobin - - * lib/pyzor/client.py: tweak the output of ping/checks/reports to - use tabs instead of ': ' as a divider - -2002-04-21 20:38 ftobin - - * lib/pyzor/: __init__.py, client.py: if we get an empty file, - don't croak - -2002-04-21 20:17 ftobin - - * NEWS, lib/pyzor/__init__.py, lib/pyzor/client.py, - lib/pyzor/server.py, scripts/pyzord: add lots of documentation, - bump version, and add CleanupAge for server. - -2002-04-21 19:13 ftobin - - * MANIFEST, scripts/.cvsignore: changes to reflect the new files - used - -2002-04-21 19:04 ftobin - - * scripts/test.sh: check for logfile existence - -2002-04-21 18:56 ftobin - - * INSTALL, lib/pyzor/__init__.py, lib/pyzor/client.py, - lib/pyzor/server.py, scripts/config, scripts/pyzord, - scripts/servers, scripts/test.sh: lots of changes, including - logging, multiple servers support, config files, user/auth with - messages - -2002-04-21 18:43 ftobin - - * scripts/.pyzor: remove this old file - -2002-04-20 21:56 ftobin - - * lib/pyzor/__init__.py: Change readyexec filenames, as per - readyexec 0.2.0 - -2002-04-17 00:40 ftobin - - * NEWS: talk about how we talk to multiple servers - -2002-04-16 20:46 ftobin - - * lib/pyzor/__init__.py, lib/pyzor/client.py, scripts/test.sh: able - to contact multiple servers - -2002-04-16 14:32 ftobin - - * NEWS: talk about how we support ReadyExec - -2002-04-16 14:17 ftobin - - * lib/pyzor/__init__.py: docs on how to use with readyexec - -2002-04-16 14:17 ftobin - - * lib/pyzor/server.py: to help xemacs with highlighting, move the - copyright stuff into a comment, instead of docstring - -2002-04-16 14:12 ftobin - - * MANIFEST: add THANKS - -2002-04-16 14:02 ftobin - - * lib/pyzor/client.py, scripts/pyzor: move most code to client.py, - so readyexec can be used - -2002-04-16 14:01 ftobin - - * scripts/test.in.mbox: add forgotten file - -2002-04-16 14:01 ftobin - - * scripts/test.sh: more graceful handling of errors - -2002-04-16 12:52 ftobin - - * INSTALL: where to find the gdbm tar.gz stuff - -2002-04-16 12:44 ftobin - - * lib/pyzor/.cvsignore: cvsignore to match .pyc - -2002-04-16 12:43 ftobin - - INSTALL: direction on how to test for gdbm - -2002-04-16 12:43 ftobin - - * scripts/test.in.mbox: forgot to add this earlier - -2002-04-16 11:56 ftobin - - * lib/pyzor/__init__.py: fix typo - - caught by Rick Macdougall <rickm@nougen.com> - -2002-04-16 11:55 ftobin - - * THANKS: add THANKS file. Rick Macdougall is first entry. - -2002-04-15 19:51 ftobin - - * setup.py: wasn't reading the lib dir correctly - -2002-04-15 19:48 ftobin - - * NEWS: note that no changes were made on the server end - -2002-04-15 19:46 ftobin - - * MANIFEST, NEWS, lib/pyzor/__init__.py, scripts/pyzor, - scripts/test.in.0, scripts/test.sh: now report by default expects a - normal rfc822 message. report now requires --mbox for reporting - entire mailboxes. - - bumped version - -2002-04-15 12:43 ftobin - - * .cvsignore, scripts/.cvsignore: seems like cvsignore files are - nice - -2002-04-14 18:23 ftobin - - * lib/pyzor/server.py, scripts/pyzor, scripts/test.sh: make sure - the database exists before querying it - -2002-04-14 18:08 ftobin - - * lib/pyzor/client.py: typo - -2002-04-14 18:02 ftobin - - * lib/pyzor/client.py: fix ip:port split problem - -2002-04-14 17:37 ftobin - - * MANIFEST: add testing files - -2002-04-14 17:36 ftobin - - * NEWS: add the NEWS - -2002-04-14 17:36 ftobin - - * README: add the README - -2002-04-14 17:33 ftobin - - * lib/pyzor/__init__.py: document the USR1 signal - -2002-04-14 17:30 ftobin - - * lib/pyzor/server.py: remove an erroneous debuging call - -2002-04-14 17:24 ftobin - - * lib/pyzor/__init__.py, scripts/pyzor: doc fixes, and exit false - on timeout - -2002-04-14 17:17 ftobin - - * scripts/pyzor: pyzor client has non-seekable input - -2002-04-14 17:10 ftobin - - * MANIFEST: add setup.py - -2002-04-14 17:04 ftobin - - * scripts/: .pyzor, test.in.0, test.sh: add some testing - -2002-04-14 16:48 ftobin - - * INSTALL: note that python gdbm module is required - -2002-04-14 16:44 ftobin - - * INSTALL: add this wayward file - -2002-04-14 16:33 ftobin - - * lib/pyzor/server.py, scripts/pyzord: add forking for server, and - fix some debugging - -2002-04-14 15:52 ftobin - - * lib/pyzor/__init__.py, lib/pyzor/client.py, lib/pyzor/server.py, - scripts/pyzor, scripts/pyzord: lots of documentation - -2002-04-14 15:19 ftobin - - * lib/pyzor/client.py, lib/pyzor/server.py, scripts/pyzor, - scripts/pyzord: work on args-handling - -2002-04-14 10:19 ftobin - - * MANIFEST, setup.py, lib/pyzor/__init__.py, lib/pyzor/client.py, - lib/pyzor/server.py: add header information and some distribution - files - -2002-04-14 10:00 ftobin - - * COPYING, lib/pyzor/__init__.py, lib/pyzor/client.py, - lib/pyzor/server.py, scripts/pyzor, scripts/pyzord: Initial - revision - -2002-04-14 10:00 ftobin - - * COPYING, lib/pyzor/__init__.py, lib/pyzor/client.py, - lib/pyzor/server.py, scripts/pyzor, scripts/pyzord: import -
[-] [+]	Changed	pyzor-0.5.0.tar.bz2/MANIFEST ^
@@ -6,6 +6,8 @@ README THANKS UPGRADING +docs/pyzor.1 +docs/pyzord.1 docs/usage.html lib/pyzor/__init__.py lib/pyzor/client.py
[-] [+]	Changed	pyzor-0.5.0.tar.bz2/NEWS ^
@@ -1,3 +1,60 @@ +Noteworthy changes in 0.5.0 +----------------------------------------------------------------- + +Note that the majority of changes in this release were contributed back +from the Debian pyzor package. + + * Man pages for pyzor and pyzord. + + * Changing back to signals for database locking, + rather than threads. It is likely that signals + will be removed again in the future, but the + existing threading changes caused problems. + + * Basic checks on the results of "discover". + + * Extended mbox support throughout the library. + + * Better handling on unknown encodings. + + * Added a --log option to log to a file. + + * Better handling of command-line options. + + * Improved error handling. + +Noteworthy changes in 0.4.x +----------------------------------------------------------------- + + * pyzor client now more gracefully handles base64 and + multipart decoding errors, so that it can be used + over an mbox. + + * pyzor client has new config file option in the [client] + section, Timeout, which specifies a timeout in seconds + for queries to come back to the client. + + * pyzord no longer daemonizes itself, and now writes + it logging to standard output. + + * The following server config options no longer have effect: + PidFile, LogFile. + + * Upped the allowed signed timestamp difference to be up to + 5 minutes (up from 3 minutes). + + * Removed the 'shutdown' command; implementation of + 'meta' commands need to be re-thought. + + * Rewrite of threads locking to access the database. + + * pyzord no longer handles USR1 signals; instead, it now + automatically reorganizes and cleans-up the database daily. + + * Client code now uses threading to catch timeouts, + rather than an alarm signal. + + Noteworthy changes in 0.4.0 -----------------------------------------------------------------
[-] [+]	Changed	pyzor-0.5.0.tar.bz2/PKG-INFO ^
@@ -1,6 +1,6 @@ Metadata-Version: 1.0 Name: pyzor -Version: 0.4.0 +Version: 0.5.0 Summary: networked spam-signature detection Home-page: http://pyzor.sourceforge.net/ Author: Frank J. Tobin
[-] [+]	Changed	pyzor-0.5.0.tar.bz2/THANKS ^
@@ -4,8 +4,9 @@ it complete and free of errors. Frank Tobin ftobin@neverending.org -Rick Macdougall rickm@nougen.com +Rick Macdougall rickm@nougen.com Colin Smith colin@archeus.plus.com Bobby Rose brose@med.wayne.edu Roman Suzi rnd@onego.ru -Robert Schiele schiele@users.sourceforge.net +Robert Schiele schiele@users.sourceforge.net +Tobias Klauser tux_edo@users.sourceforge.net
	Added	pyzor-0.5.0.tar.bz2/docs/._pyzor.1 ^
	Added	pyzor-0.5.0.tar.bz2/docs/._pyzord.1 ^
[-] [+]	Added	pyzor-0.5.0.tar.bz2/docs/pyzor.1 ^
@@ -0,0 +1,128 @@ +.TH PYZOR 1 "10 Oct 2002" +.SH NAME +pyzor \- spam\-catcher using a collaborative filtering network +.SH SYNOPSIS +\fBpyzor\fP [\fB\-d\fP] [\fB\-\-homedir\fP \fIdir\fP] \fIcommand\fP [\fIcommand\_options\fP] +.SH OPTIONS +.TP +\fB\-d\fP +turn on debugging +.TP +\fB\-\-homedir\fI dir\fP +use dir as the home directory for Pyzor instead of the default ~/.pyzor. +See the files section for more information on what files are inside of the +homedir. +.SH COMMANDS +.TP +\fBcheck\fP[\fB\-\-mbox\fP] +Reads on standard input an RFC 822 (email) message. Exit code is zero (0) +if and only if a match is found and the global whitelist count is zero. +.BR +If \-\-mbox is provided, then the input is assumed to be a unix mailbox, and +all messages in it will be checked. +.BR +If multiple servers are listed in the configuration file, the exit code will +be zero (0) if and only if there is a match found on at least one server +(without it being whitelisted anyplace). +.TP +\fBreport \fP[\fB\-\-mbox\fP] +Reads on standard input an RFC 822 (email) message. Reports to the server a +digest of each message in the mailbox as spam. Writes to standard output a +tuple of (error\-code, message) from the server. +.BR +If \-\-mbox is provided, then the input is assumed to be a unix mailbox, and +all messages in it will be sent to the server. +.TP +\fBwhitelist \fP[\fB\-\-mbox\fP] +Reads on standard input an RFC 822 (email) message. Sends to the server a +digest of each message in the mailbox for whitelisting. Writes to standard +output a tuple of (error\-code, message) from the server. +.BR +If \-\-mbox is provided, then the input is assumed to be a unix mailbox, and +all messages in it will be sent to the server. +.TP +\fBdiscover\fP +Finds Pyzor servers, and writes them to ~/.pyzor/servers. This may +accomplished through querying already-known servers or an HTTP call to a +hard-coded address. +.TP +\fBping\fP +Merely requests a response from the servers. +.TP +\fBgenkey\fP +Based upon a secret passphrase gathered from the user and salt gathered from +/dev/random, prints to standard output a tuple of "salt,key". +Used to put account information into the accounts file. +See the section Using Accounts for more information. +.TP +\fBdigest \fP[\fB\-\-mbox\fP] +Reads on standard input an RFC 822 (email) message. Writes the +digest of the message to standard output. +.BR +If \-\-mbox is provided, then the input is assumed to be a unix +mailbox, each message's digest is written to standard output, separated +by newlines. +.TP +\fBpredigest\fP +Reads on standard input an RFC 822 (email) message. Writes to +standard output the normalized lines of data that are digested, with the +exception that the lines printed have newlines (all whitespace is +removed before digesting). + +.SH USING PYZOR WITH READYEXEC +\fBReadyExec\fP is a system to eliminate the high startup-cost of executing +scripts repeatedly. If you execute pyzor a lot, you might be interested +in installing ReadyExec and using it with pyzor. + +To use pyzor with ReadyExec, the readyexecd.py server needs to be +started as: + + readyexecd.py socket_file pyzor + +socket_file can be any (non\-existing) filename you wish ReadyExec to +use, such as /tmp/pyzor: + + readyexecd.py /tmp/pyzor pyzor + +Individual clients are then executed as: + + readyexec socket_file options command cmd_options + +For example: + + readyexec /tmp/pyzor check + readyexec /tmp/pyzor report + readyexec /tmp/pyzor whitelist \-\-mbox + readyexec /tmp/pyzor \-d ping + +ReadyExec can be found at: http://readyexec.sourceforge.net/ +.SH INTEGRATION WITH MUTT +Add the following line to mutt.conf: + + macro index S "\|/usr/bin/pyzor report" + +Then press S on the spam message in mutt to report it with pyzor. +.SH FILES +\fI~/.pyzor/config\fP + +The format of this file is INI-style (name=value, divided into [sections]). +Names are case insensitive. All values which are filenames can have +shell\-style tildes (~) in them. All values which are relative filenames are +interpreted to be relative to the Pyzor homedir. + +\fBDefaults\fP + [client] + ServersFile = servers + AccountsFile = accounts + DiscoverServersURL = http://pyzor.sourceforge.net/cgi-bin/inform\-servers\-0\-3\-x + Timeout = 5 +.SH SEE ALSO +pyzord(1) +.SH AUTHOR +This manpage was originally written by Bastian Kleineidam +<calvin@debian.org> for the Debian distribution of pyzor but may be used +by others. +.BR +The main author of pyzor is Frank J. Tobin <ftobin@neverending.org>. The +main project page for pyzor can be found at +http://sourceforge.net/projects/pyzor
[-] [+]	Added	pyzor-0.5.0.tar.bz2/docs/pyzord.1 ^
@@ -0,0 +1,44 @@ +.TH PYZORD 1 "10 Oct 2002" +.SH NAME +pyzord \- spam\-catching server +.SH SYNOPSIS +\fBpyzord\fP [\fB\-d\fP] [\fB\-\-homedir\fP \fIdir\fP] + +Note: pyzord does not daemonize itself. + +Note: logging information is written to standard output. + +.SH OPTIONS +.TP +\fB\-d\fP +Turn on debugging +.TP +\fB\-\-homedir\fP \fIdir\fP +use dir as the home directory for Pyzor instead of the default ~/.pyzor. +See the files section for more information on what files are inside of the +homedir. +.SH FILES +\fI~/.pyzor/config\fP + +The format of this file is INI-style (name=value, divided into [sections]). +Names are case insensitive. All values which are filenames can have +shell\-style tildes (~) in them. All values which are relative filenames are +interpreted to be relative to the Pyzor homedir. + +\fBDefaults\fP + [server] + Port = 24441 + ListenAddress = 0.0.0.0 + DigestDB = pyzord.db + PasswdFile = pyzord.passwd + AccessFile = pyzord.access +.SH SEE ALSO +pyzor(1) +.SH AUTHOR +This manpage was originally written by Bastian Kleineidam +<calvin@debian.org> for the Debian distribution of pyzor but may be used +by others. +.BR +The main author of pyzor is Frank J. Tobin <ftobin@neverending.org>. The +main project page for pyzor can be found at +http://sourceforge.net/projects/pyzor
[-] [+]	Changed	pyzor-0.5.0.tar.bz2/docs/usage.html ^
@@ -9,8 +9,8 @@ <body> <h1>Pyzor Usage Documentation</h1> - <p id="cvsid">$Id: usage.html,v 1.7 2002/09/08 03:33:44 ftobin Exp $</p> - + <p id="cvsid">$Id: usage.html,v 1.11 2002-10-10 21:37:08 ftobin Exp $</p> + <ul> <li><a href="#client"><kbd>pyzor</kbd> (client)</a></li> <li><a href="#server"><kbd>pyzord</kbd> (server)</a></li> @@ -21,17 +21,17 @@ </ul> <h2 id="client"><kbd>pyzor</kbd> (client)</h2> - + <pre> pyzor [-d] [--homedir <var class="filename">dir</var>] <var>command</var> [<var>command_options</var>] </pre> - + <h3 id="options">options</h3> - + <dl> <dt><kbd>-d</kbd></dt> <dd>turn on debugging</dd> - + <dt id="pyzor-homedir"><kbd>--homedir <var class="filename">dir</var></kbd></dt> <dd> use <var>dir</var> as the home directory @@ -41,13 +41,13 @@ more information on what files are inside of the homedir. </dd> - + </dl> - + <h3 id="commands">commands</h3> <h4>General Output</h4> - + <p> In general, the output from <kbd>pyzor <var>command</var></kbd> is of the form: @@ -73,13 +73,13 @@ is a match found on at least one server (without it being whitelisted anyplace). </p> - + <p> The command-specfic output for a <kbd>check</kbd> is <samp><var>report-count</var> <var>whitelist-count</var></samp>. </p> </dd> - + <dt id="report"><kbd>report [--mbox]</kbd></dt> <dd> <p> @@ -94,7 +94,7 @@ will be sent to the server. </p> </dd> - + <dt id="whitelist"><kbd>whitelist [--mbox]</kbd></dt> <dd> <p> @@ -109,7 +109,7 @@ will be sent to the server. </p> </dd> - + <dt id="discover"><kbd>discover</kbd></dt> <dd> <p> @@ -119,19 +119,11 @@ servers or an HTTP call to a hard-coded address. </p> </dd> - + <dt id="ping"><kbd>ping</kbd></dt> <dd> <p>Merely requests a response from the servers.</p> </dd> - - <dt id="shutdown"><kbd>shutdown <var>server ...</var></kbd></dt> - <dd> - <p> - Requests each <kbd><var>server</var></kbd> to be shutdown. - Format for <kbd><var>server</var></kbd> is <kbd>host:port</kbd>. - </p> - </dd> <dt id="genkey"><kbd>genkey</kbd></dt> <dd> @@ -143,7 +135,7 @@ See the section <a href="#using-accounts">Using Accounts</a> for more information. </dd> - + <dt id="digest"><kbd>digest [--mbox]</kbd></dt> <dd> <p> @@ -169,10 +161,10 @@ </dd> </dl> - - + + <h3>Using the Pyzor client with procmail</h3> - + <p> To use Pyzor in a procmail system, consider using the following simple recipe. @@ -190,7 +182,7 @@ marked with Pyzor, instead of immediately filtering them into a separate folder: </p> - + <pre> :0 Wc \| pyzor check @@ -200,7 +192,7 @@ <h2>Using the Pyzor client with <a href="http://readyexec.sourceforge.net/">ReadyExec</a></h2> - + <p> <a href="http://readyexec.sourceforge.net/">ReadyExec</a> is a system to eliminate the high startup-cost @@ -208,12 +200,12 @@ <kbd>pyzor</kbd> a lot, you might be interested in installing ReadyExec and using it with <kbd>pyzor</kbd>. </p> - + <p> To use pyzor with ReadyExec, the readyexecd.py server needs to be started as: </p> - + <pre> readyexecd.py <var>socket_file</var> pyzor.client.run </pre> @@ -227,19 +219,19 @@ <pre> readyexecd.py /tmp/pyzor pyzor.client.run </pre> - + <p> Individual clients are then executed as: </p> - + <pre> readyexec <var>socket_file</var> <var>options</var> <var>command</var> <var>cmd_options</var> </pre> - + <p> For example: </p> - + <pre> readyexec /tmp/pyzor check readyexec /tmp/pyzor report @@ -249,17 +241,25 @@ <h2 id="server"><kbd>pyzord</kbd> (server)</h2> - + <pre> pyzord [-d] [--homedir <var>dir</var>] </pre> - + + <p> + Note: pyzord does not daemonize itself. + </p> + + <p> + Note: logging information is written to standard output. + </p> + <h3>options</h3> - + <dl> <dt><kbd>-d</kbd></dt> - <dd>Turn on debugging</dd> - + <dd>Turn on debugging (writes information to standard error)</dd> + <dt id="pyzord-homedir"><kbd>--homedir <var class="filename">dir</var></kbd></dt> <dd> use <var>dir</var> as the home directory @@ -269,27 +269,13 @@ more information on what files are inside of the homedir. </dd> - - </dl> - - <h3>Signal handling</h3> - <dl> - <dt>HUP</dt> - <dd> - The server will re-open the logfile. - </dd> - <dt>USR1</dt> - <dd> - The database is 'cleaned up'. - </dd> </dl> - <h2 id="files">Files</h2> - + <h3 id="config"><kbd>~/pyzor/config</kbd></h3> - + <p> The format of this file is INI-style (<kbd><var>name</var>=<var>value</var></kbd>, @@ -300,27 +286,26 @@ filenames are interpreted to be relative to the Pyzor <a href="#pyzor-homedir">homedir</a>. </p> - + <h4>Defaults</h4> - + <pre> <a href="#section-client">[client]</a> <a href="#ServersFile">ServersFile</a> = servers <a href="#AccountsFile">AccountsFile</a> = accounts <a href="#DiscoverServersURL">DiscoverServersURL</a> = http://pyzor.sourceforge.net/cgi-bin/inform-servers-0-3-x +<a href="#Timeout">Timeout</a> = 5 <a href="#section-server">[server]</a> <a href="#Port">Port</a> = 24441 <a href="#ListenAddress">ListenAddress</a> = 0.0.0.0 -<a href="#LogFile">LogFile</a> = pyzord.log -<a href="#PidFile">PidFile</a> = pyzord.pid <a href="#DigestDB">DigestDB</a> = pyzord.db <a href="#PasswdFile">PasswdFile</a> = pyzord.passwd <a href="#AccessFile">AccessFile</a> = pyzord.access </pre> - + <h4>Definitions</h4> - + <h5 id="section-client"><kbd>[client]</kbd> section</h5> <dl> @@ -331,24 +316,24 @@ <kbd>host:port</kbd>. Can be populated with <kbd>pyzor discover</kbd>. </dd> - + <dt id="AccountsFile">AccountsFile</dt> <dd> <p> File containing information about accounts on servers. Format is line-oriented, with each line being: </p> - + <pre> <var>host</var> : <var>port</var> : <var>username</var> : <var>salt,key</var> </pre> - + <p>Example:</p> - + <pre> 127.0.0.1 : 9999 : bob : 227bfb58efaba7c582d9dcb66ab2063d38df2923,8da9f54058c34e383e997f45d6eb74837139f83b </pre> - + <p> See the section <a href="#using-accounts">Using Accounts</a> for more information. @@ -360,45 +345,44 @@ During <kbd>pyzor discover</kbd>, the URL to use in finding servers. </dd> - + + <dt id="Timeout">Timeout</dt> + <dd> + Number of seconds to wait for response to a query. + </dd> + </dl> - + <h5 id="section-server"><kbd>[server]</kbd> section</h5> <dl> <dt id="Port">Port</dt> <dd>port to listen on </dd> - + <dt id="ListenAddress">ListenAddress</dt> <dd>address to listen on</dd> - - <dt id="PidFile">PidFile</dt> - <dd>file to contain the process id of the server</dd> - - <dt id="LogFile">LogFile</dt> - <dd>file to contain server logs</dd> - + <dt id="DigestDB">DigestDB</dt> <dd>file containing the database of digests</dd> - + <dt id="PasswdFile">PasswdFile</dt> <dd> <p> File containing a list of user account information. Line format: </p> - + <pre> <var>username</var> : <var>key</var> </pre> - + <p>Example:</p> <pre> bob : 8da9f54058c34e383e997f45d6eb74837139f83b </pre> - + </dd> - + <dt id="AccessFile">AccessFile</dt> <dd> <p> @@ -409,7 +393,7 @@ <pre> <var>privilege ...</var> : <var>username ...</var> : allow\|deny </pre> - + <dl> <dt><var>privilege ...</var></dt> <dd> @@ -419,62 +403,62 @@ The keyword <kbd>all</kbd> can be used to to refer to all commands. </dd> - + <dt><var>username ...</var></dt> <dd> a list of whitespace-separated usernames. - The keyword <kbd>all</kbd> can be used to refer to + The keyword <kbd>all</kbd> can be used to refer to all users. The anonymous user is refereed to as <kbd>anonymous</kbd>. </dd> - + <dt>allow\|deny</dt> <dd> Whether or not the specified user(s) can perform the specified privilege(s) on the line. </dd> - + </dl> - + <p> The file is processed from top to bottom, with the first match for user/privilege being the value taken. Every file has the following implicit final rule: </p> - + <pre> all : all : deny </pre> - + <p> If this file is nonexistant, the following default is used: </p> - + <pre> check report ping info : anonymous : allow </pre> - + </dd> - + </dl> - + <h2 id="using-accounts">Using Accounts</h2> - + <p> To get an account on a server requires coordination between the client user and server admin. Use the following steps: </p> - + <ol> <li> User and admin should agree on a username for the user. - Allowed characters for a username are alpha-numerics, + Allowed characters for a username are alpha-numerics, the underscore, and dashes. The normative regular expression it must match is <kbd>^[-\.\w]+$</kbd>. Let us assume they have agreed on 'bob'. </li> - + <li> <p> User generates a key with <kbd>pyzor genkey</kbd>, and @@ -490,13 +474,13 @@ 127.0.0.1 : 9999 : bob : 227bfb58efaba7c582d9dcb66ab2063d38df2923,8da9f54058c34e383e997f45d6eb74837139f83b </pre> </li> - + <li> The user then sends the key (<em>the part to the <strong>right</strong>-hand side of the comma</em>) to the admin. </li> - + <li> <p> The admin adds the following to their @@ -514,29 +498,29 @@ the admin then adds the appropriate permissions to <span class="filename">~/.pyzor/pyzord.access</span>: </p> - + <pre> check report ping info whitelist : bob : allow </pre> - + </li> - + <li> The server needs to be restarted in order for this to take effect. </li> - + </ol> - + <h2 id="razor-diff">User-level Differences from Razor</h2> - + <p> This is a small list of user-recognizable differences between Pyzor and Razor that one might notice if coming from Razor to Pyzor. </p> - + <ul> <li> Pyzor does not consult a whitelist system for you @@ -549,9 +533,9 @@ </ul> <h2 id="copyright">Copyright</h2> - + <p> - Copyright © 2002 Frank J. Tobin + Copyright © 2002-9 Frank J. Tobin <a href="mailto:ftobin@neverending.org">ftobin@neverending.org</a> </p> @@ -576,6 +560,6 @@ along with this program; if not, visit the following URL: <a href="http://www.gnu.org/copyleft/gpl.html">http://www.gnu.org/copyleft/gpl.html</a>. </p> - + </body> </html>
[-] [+]	Changed	pyzor-0.5.0.tar.bz2/lib/pyzor/__init__.py ^
@@ -1,8 +1,8 @@ """networked spam-signature detection""" __author__ = "Frank J. Tobin, ftobin@neverending.org" -__version__ = "0.4.0" -__revision__ = "$Id: __init__.py,v 1.41 2002/09/07 01:12:12 ftobin Exp $" +__version__ = "0.5.0" +__revision__ = "$Id: __init__.py,v 1.43 2002-09-17 15:12:58 ftobin Exp $" import os import os.path @@ -75,7 +75,7 @@ class Username(str): user_pattern = re.compile(r'^[-\.\w]+$') - + def __init__(self, s): self.validate() @@ -87,7 +87,7 @@ class Opname(str): op_pattern = re.compile(r'^[-\.\w]+$') - + def __init__(self, s): self.validate() @@ -106,15 +106,15 @@ def data(self, msg): print msg def warn(self, msg): - if not self.quiet: sys.__stderr__.write('%s\n' % msg) + if not self.quiet: sys.stderr.write('%s\n' % msg) def debug(self, msg): - if self.do_debug: sys.__stderr__.write('%s\n' % msg) + if self.do_debug: sys.stderr.write('%s\n' % msg) class DataDigest(str): # hex output doubles digest size - value_size = sha.digest_size * 2 + value_size = sha.digest_size * 2 def __init__(self, value): if len(value) != self.value_size: @@ -135,7 +135,7 @@ raise ValueError, "offset percentage out of bounds" if not length > 0: raise ValueError, "piece lengths must be positive" - + validate_tuple = staticmethod(validate_tuple) def netstring(self): @@ -155,7 +155,7 @@ self.validate_tuple(perc_offset, length) new_spec.append((perc_offset, length)) - + return new_spec from_netstring = classmethod(from_netstring) @@ -166,7 +166,7 @@ def __init__(self, fp=None): if fp is None: fp = cStringIO.StringIO() - + super(Message, self).__init__(fp) self.setup() @@ -207,13 +207,13 @@ self.setdefault('PV', str(proto_version)) super(ThreadedMessage, self).init_for_sending() - + def ensure_complete(self): if not (self.has_key('PV') and self.has_key('Thread')): raise IncompleteMessageError, \ "doesn't have fields for a ThreadedMessage" super(ThreadedMessage, self).ensure_complete() - + def get_protocol_version(self): return float(self['PV']) @@ -227,8 +227,8 @@ class MacEnvelope(Message): - ts_diff_max = 180 - + ts_diff_max = 300 + def ensure_complete(self): if not (self.has_key('User') and self.has_key('Time') @@ -240,15 +240,15 @@ def get_submsg(self, factory=ThreadedMessage): self.rewindbody() return apply(factory, (self.fp,)) - + def verify_sig(self, user_key): typecheck(user_key, long) - + user = Username(self['User']) ts = int(self['Time']) said_sig = self['Sig'] hashed_user_key = self.hash_key(user_key, user) - + if abs(time.time() - ts) > self.ts_diff_max: raise SignatureError, "timestamp not within allowed range" @@ -261,7 +261,7 @@ def wrap(self, user, key, msg): """This should be used to create a MacEnvelope""" - + typecheck(user, str) typecheck(msg, Message) typecheck(key, long) @@ -294,9 +294,9 @@ """returns lower(H(U + ':' + lower(hex(K))))""" typecheck(key, long) typecheck(user, Username) - + return sha.new("%s:%x" % (Username, key)).hexdigest().lower() - + hash_key = staticmethod(hash_key) @@ -307,9 +307,9 @@ M is message T is decimal epoch timestamp K is hashed_key - + returns a digest object""" - + typecheck(ts, int) typecheck(msg, Message) typecheck(hashed_key, str) @@ -317,7 +317,7 @@ h_msg = self.hash_msg(msg) return sha.new("%s:%d:%s" % (h_msg.digest(), ts, hashed_key)).hexdigest().lower() - + sign_msg = classmethod(sign_msg) @@ -349,7 +349,7 @@ """this is class that should be used to read in Requests of any type. subclasses are responsible for setting 'Op' if they are generating a message""" - + def get_op(self): return self['Op'] @@ -365,7 +365,7 @@ def setup(self): super(Request, self).setup() self.setdefault('Op', self.op) - + class PingRequest(ClientSideRequest): @@ -412,7 +412,7 @@ def __init__(self, code, s): typecheck(code, int) typecheck(s, str) - + super(ErrorResponse, self).__init__() self.setdefault('Code', str(code)) self.setdefault('Diag', s) @@ -424,7 +424,7 @@ full_range = (0, 2**16) ok_range = (1024, full_range[1]) error_value = 0 - + def __init__(self, i): super(ThreadId, self).__init__(i) if not (self.full_range[0] <= self < self.full_range[1]): @@ -448,7 +448,7 @@ typecheck(self[1], int) if len(self) != 2: raise ValueError, "invalid address: %s" % str(self) - + def __str__(self): return (self[0] + ':' + str(self[1])) @@ -468,7 +468,7 @@ assert isinstance(homedir, str) self.homedir = homedir super(Config, self).__init__() - + def get_filename(self, section, option): fn = os.path.expanduser(self.get(section, option)) if not os.path.isabs(fn): @@ -478,14 +478,14 @@ def get_homedir(specified): homedir = os.path.join('/etc', 'pyzor') - + if specified is not None: homedir = specified else: userhome = os.getenv('HOME') if userhome is not None: homedir = os.path.join(userhome, '.pyzor') - + return homedir @@ -501,15 +501,16 @@ dict.""" if kwargs is None: kwargs = {} - + saved = {} for (k, v) in repl.items(): saved[k] = globs[k] globs[k] = v - r = apply(obj, varargs, kwargs) - - globs.update(saved) + try: + r = apply(obj, varargs, kwargs) + finally: + globs.update(saved) return r
[-] [+]	Changed	pyzor-0.5.0.tar.bz2/lib/pyzor/client.py ^
@@ -1,5 +1,6 @@ """networked spam-signature detection client""" +import re import os import os.path import socket @@ -8,6 +9,7 @@ import getopt import tempfile import mimetools +import multifile import sha import pyzor @@ -15,20 +17,18 @@ __author__ = pyzor.__author__ __version__ = pyzor.__version__ -__revision__ = "$Id: client.py,v 1.41 2002/09/08 20:37:15 ftobin Exp $" +__revision__ = "$Id: client.py,v 1.48 2003-02-01 10:29:42 ftobin Exp $" randfile = '/dev/random' class Client(object): __slots__ = ['socket', 'output', 'accounts'] - ttl = 4 timeout = 5 max_packet_size = 8192 - + def __init__(self, accounts): signal.signal(signal.SIGALRM, handle_timeout) - self.accounts = accounts self.output = Output() self.build_socket() @@ -42,7 +42,7 @@ msg = InfoRequest(digest) self.send(msg, address) return self.read_response(msg.get_thread()) - + def report(self, digest, spec, address): msg = ReportRequest(digest, spec) self.send(msg, address) @@ -58,18 +58,13 @@ self.send(msg, address) return self.read_response(msg.get_thread()) - def shutdown(self, address): - msg = ShutdownRequest() - self.send(msg, address) - return self.read_response(msg.get_thread()) - def build_socket(self): self.socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) def send(self, msg, address): msg.init_for_sending() account = self.accounts[address] - + mac_msg_str = str(MacEnvelope.wrap(account.username, account.keystuff.key, msg)) @@ -114,25 +109,34 @@ class ServerList(list): inform_url = 'http://pyzor.sourceforge.net/cgi-bin/inform-servers-0-3-x' - + def read(self, serverfile): for line in serverfile: orig_line = line line = line.strip() - if line and not line.startswith('#'): + if line and not line.startswith('#') and \ + re.match('[a-zA-Z0-9.-]+:[0-9]+', line): self.append(pyzor.Address.from_str(line)) class ExecCall(object): __slots__ = ['client', 'servers', 'output'] - + # hard-coded for the moment digest_spec = DataDigestSpec([(20, 3), (60, 3)]) def run(self): debug = 0 - (options, args) = getopt.getopt(sys.argv[1:], 'dh:', ['homedir=']) + log = None + options = None + + try: + (options, args) = getopt.getopt(sys.argv[1:], 'dh:', + ['homedir=', 'log', 'help']) + except getopt.GetoptError: + self.usage() + if len(args) < 1: self.usage() @@ -141,30 +145,37 @@ for (o, v) in options: if o == '-d': debug = 1 - elif o == '-h': + elif o in ('-h', '--help'): self.usage() elif o == '--homedir': specified_homedir = v - - self.output = Output(debug=debug) + elif o == '--log': + log = 1 + self.output = Output(debug=debug) homedir = pyzor.get_homedir(specified_homedir) + if log: + sys.stderr = open(homedir + "/pyzor.log", 'a') + sys.stderr.write("\npyzor[" + repr (os.getpid()) + "]:\n") + config = pyzor.Config(homedir) config.add_section('client') - defaults = {'ServersFile': 'servers', + defaults = {'ServersFile': 'servers', 'DiscoverServersURL': ServerList.inform_url, - 'AccountsFile' : 'accounts', + 'AccountsFile': 'accounts', + 'Timeout': str(Client.timeout), } for k, v in defaults.items(): config.set('client', k, v) - + config.read(os.path.join(homedir, 'config')) - + servers_fn = config.get_filename('client', 'ServersFile') - + Client.timeout = config.getint('client', 'Timeout') + if not os.path.exists(homedir): os.mkdir(homedir) @@ -176,6 +187,12 @@ self.servers = self.get_servers(servers_fn) + if not len(self.servers): + sys.stderr.write("no valid servers found\n") + # Remove the servers file as it only contains invalid entries, + # probably a HTTP error message. + os.remove(servers_fn) + sys.exit(1) self.client = Client(self.get_accounts(config.get_filename('client', 'AccountsFile'))) @@ -197,10 +214,13 @@ def usage(self, s=None): if s is not None: sys.stderr.write("%s\n" % s) - sys.stderr.write("""usage: %s [-d] [--homedir dir] command [cmd_opts] + sys.stderr.write(""" +usage: %s [-d] [--homedir dir] command [cmd_opts] command is one of: check, report, discover, ping, digest, predigest, - genkey, shutdown + genkey + Data is read on standard input (stdin). + """ % sys.argv[0]) sys.exit(2) @@ -208,9 +228,9 @@ def ping(self, args): - getopt.getopt(args[1:], '') - - if len(args) > 1: + try: + getopt.getopt(args[1:], '') + except getopt.GetoptError: self.usage("%s does not take any non-option arguments" % args[0]) runner = ClientRunner(self.client.ping) @@ -219,32 +239,25 @@ runner.run(server, (server,)) return runner.all_ok - - def shutdown(self, args): - (opts, args2) = getopt.getopt(args[1:], '') - if len(args2) > 1: + def info(self, args): + try: + (options, args2) = getopt.getopt(args[1:], '', ['mbox']) + except getopt.GetoptError: self.usage("%s does not take any non-option arguments" % args[0]) - runner = ClientRunner(self.client.shutdown) - - for arg in args2: - server = Address.from_str(arg) - runner.run(server, (server,)) - - return runner.all_ok - + do_mbox = 'msg' - def info(self, args): - getopt.getopt(args[1:], '') - - if len(args) > 1: - self.usage("%s does not take any non-option arguments" % args[0]) + for (o, v) in options: + if o == '--mbox': + do_mbox = 'mbox' runner = InfoClientRunner(self.client.info) - for digest in FileDigester(sys.stdin, self.digest_spec): + for digest in get_input_handler(sys.stdin, self.digest_spec, do_mbox): + if not digest: + continue for server in self.servers: response = runner.run(server, (digest, server)) @@ -252,38 +265,49 @@ def check(self, args): - getopt.getopt(args[1:], '') - - if len(args) > 1: + try: + (options, args2) = getopt.getopt(args[1:], '', ['mbox']) + except getopt.GetoptError: self.usage("%s does not take any non-option arguments" % args[0]) + do_mbox = 'msg' + + for (o, v) in options: + if o == '--mbox': + do_mbox = 'mbox' + runner = CheckClientRunner(self.client.check) - for digest in FileDigester(sys.stdin, self.digest_spec): + for digest in get_input_handler(sys.stdin, self.digest_spec, do_mbox): + if not digest: + continue for server in self.servers: - response = runner.run(server, (digest, server)) - + runner.run(server, (digest, server)) + return (runner.found_hit and not runner.whitelisted) def report(self, args): - (options, args2) = getopt.getopt(args[1:], '', ['mbox']) - do_mbox = False - - if len(args2) > 1: + try: + (options, args2) = getopt.getopt(args[1:], '', ['mbox']) + except getopt.GetoptError: self.usage("%s does not take any non-option arguments" % args[0]) + do_mbox = 'msg' + for (o, v) in options: if o == '--mbox': - do_mbox = True - + do_mbox = "mbox" + all_ok = True - for digest in FileDigester(sys.stdin, self.digest_spec, do_mbox): + for digest in get_input_handler(sys.stdin, self.digest_spec, do_mbox): + if not digest: + continue if not self.send_digest(digest, self.digest_spec, self.client.report): all_ok = False - + return all_ok @@ -297,59 +321,64 @@ for server in self.servers: runner.run(server, (digest, spec, server)) - + return runner.all_ok def whitelist(self, args): - (options, args2) = getopt.getopt(args[1:], '', ['mbox']) - - if len(args2) > 1: + try: + (options, args2) = getopt.getopt(args[1:], '', ['mbox']) + except getopt.GetoptError: self.usage("%s does not take any non-option arguments" % args[0]) - do_mbox = False + do_mbox = "msg" for (o, v) in options: if o == '--mbox': - do_mbox = True - + do_mbox = "mbox" + all_ok = True - for digest in FileDigester(sys.stdin, self.digest_spec, do_mbox): + for digest in get_input_handler(sys.stdin, self.digest_spec, do_mbox): + if not digest: + continue if not self.send_digest(digest, self.digest_spec, self.client.whitelist): all_ok = False - + return all_ok def digest(self, args): - (options, args2) = getopt.getopt(args[1:], '', ['mbox']) - - if len(args2) > 1: + try: + (options, args2) = getopt.getopt(args[1:], '', ['mbox']) + except getopt.GetoptError: self.usage("%s does not take any non-option arguments" % args[0]) - - do_mbox = False + do_mbox = "msg" for (o, v) in options: if o == '--mbox': - do_mbox = True - - for digest in FileDigester(sys.stdin, self.digest_spec, do_mbox): + do_mbox = "mbox" + + for digest in get_input_handler(sys.stdin, self.digest_spec, do_mbox): + if not digest: + continue sys.stdout.write("%s\n" % digest) return True def print_digested(self, args): - getopt.getopt(args[1:], '') - - if len(args) > 1: + try: + getopt.getopt(args[1:], '') + except getopt.GetoptError: self.usage("%s does not take any non-option arguments" % args[0]) + do_mbox = "msg" + def loop(): - for digest in FileDigester(sys.stdin, self.digest_spec): + for digest in get_input_handler(sys.stdin, self.digest_spec, do_mbox): pass modglobal_apply(globals(), {'DataDigester': PrintingDataDigester}, @@ -358,9 +387,9 @@ return True def genkey(self, args): - getopt.getopt(args[1:], '') - - if len(args) > 1: + try: + getopt.getopt(args[1:], '') + except getopt.GetoptError: self.usage("%s does not take any non-option arguments" % args[0]) import getpass @@ -414,7 +443,6 @@ 'report': report, 'ping' : ping, 'genkey': genkey, - 'shutdown': shutdown, 'info': info, 'whitelist': whitelist, 'digest': digest, @@ -428,7 +456,7 @@ """The major workhouse class""" __slots__ = ['_atomic', '_value', '_used_line', '_digest', 'seekable'] - + # minimum line length for it to be included as part # of the digest. I forget the purpose, however. # Someone remind me so I can document it here. @@ -465,8 +493,8 @@ self.seekable = seekable (fp, offsets) = self.get_line_offsets(fp) - - # did we get an empty file? + + # did we get an empty (parsed output) file? if len(offsets) == 0: return @@ -486,9 +514,12 @@ def handle_atomic(self, fp): """we digest everything""" self._atomic = True - fp.seek(0) - for line in fp: - self.handle_line(line) + try: + fp.seek(0) + for line in fp: + self.handle_line(line) + except: + pass def handle_pieced(self, fp, spec, offsets): @@ -519,7 +550,7 @@ cur_offset = fp.tell() newfp = None else: - # we need a seekable file because to make + # we need a seekable file to make # line-based skipping around to be more efficient # than loading the whole thing into memory cur_offset = 0 @@ -527,7 +558,7 @@ offsets = [] - + for line in fp: norm = self.normalize(line) should_handle = self.should_handle_line(norm) @@ -583,9 +614,9 @@ def normalize(self, s): repl = self.unwanted_txt_repl s2 = s + s2 = self.longstr_ptrn.sub(repl, s2) s2 = self.email_ptrn.sub(repl, s2) s2 = self.url_ptrn.sub(repl, s2) - s2 = self.longstr_ptrn.sub(repl, s2) s2 = self.html_tag_ptrn.sub(repl, s2) # make sure we do the whitespace last because some of # the previous patterns rely on whitespace @@ -608,35 +639,41 @@ -class FileDigester(BasicIterator): - __slots__ = ['digester'] +def get_input_handler(fp, spec, style='msg', seekable=False): + """Return an object that can be iterated over + to get all the digests from fp according to spec. + mbox is a boolean""" + if style == 'msg': + return filter(lambda x: x is not None, + (DataDigester(rfc822BodyCleaner(fp), + spec, seekable).get_digest(),) + ) - def __init__(self, fp, spec, mbox=False): - self.digester = iter(get_file_digester(fp, spec, mbox)) - self.output = pyzor.Output() + elif style =='mbox': + return MailboxDigester(fp, spec) - def next(self): - digest = self.digester.next() - self.output.debug("calculated digest: %s" % digest) - return digest + elif style == 'digests': + return JustDigestsIterator(fp) + raise ValueError, "unknown input style" -def get_file_digester(fp, spec, mbox, seekable=False): - """Return an object that can be iterated over - to get all the digests from fp according to spec. - mbox is a boolean""" - if mbox: - return MailboxDigester(fp, spec) +class JustDigestsIterator(BasicIterator): + __slots__ = ['fp'] - return (DataDigester(rfc822BodyCleaner(fp), - spec, seekable).get_digest(),) + def __init__(self, fp): + self.fp = fp + def next(self): + l = fp.readline() + if not l: + raise StopIteration + return l.rstrip() class MailboxDigester(BasicIterator): __slots__ = ['mbox', 'digest_spec', 'seekable'] - + def __init__(self, fp, digest_spec, seekable=False): import mailbox self.mbox = mailbox.PortableUnixMailbox(fp, @@ -645,7 +682,12 @@ self.seekable = seekable def next(self): - next_msg = self.mbox.next() + try: + next_msg = self.mbox.next() + except IOError: + print "Error: Please feed mailbox files in on stdin, i.e." + print " pyzor digest --mbox < my_mbox_file" + next_msg = None if next_msg is None: raise StopIteration return DataDigester(next_msg, self.digest_spec, @@ -655,64 +697,110 @@ class rfc822BodyCleaner(BasicIterator): __slots__ = ['fp', 'multifile', 'curfile', 'type'] - + def __init__(self, fp): msg = mimetools.Message(fp, seekable=0) - self.type = msg.getmaintype() + # Default type is text. See #1529694. + self.type = msg.getmaintype() or "text" self.multifile = None self.curfile = None + # Check if we got a mail or not. Set type to binary if there is no + # 'From:' header and type is text/plain with encoding 7bit. + # 7bit is passed through anyway so nobody cares. + if (not msg.has_key("From") and self.type == 'text' and + msg.subtype == 'plain' and msg.getencoding() == '7bit'): + self.type = 'binary' + + if self.type is '': + self.type = 'text' + if self.type == 'text': encoding = msg.getencoding() - if encoding == '7bit': - self.curfile = msg.fp - else: + self.curfile = msg.fp + if encoding != '7bit': + # fix bad encoding name + if encoding == '8bits': + encoding = '8bit' + + import binascii self.curfile = tempfile.TemporaryFile() - mimetools.decode(msg.fp, self.curfile, encoding) - self.curfile.seek(0) - + try: + mimetools.decode(msg.fp, self.curfile, encoding) + except binascii.Error, e: + sys.stderr.write("%s: %s\n" % (e.__class__, e)) + self.curfile = cStringIO.StringIO() + except ValueError, e: + #sys.stderr.write("%s: %s\n" % (e.__class__, e)) + self.curfile = msg.fp + except multifile.Error, e: + #sys.stderr.write("%s: %s\n" % (e.__class__, e)) + self.curfile = msg.fp + try: + self.curfile.seek(0) + except: + # If we get an error we'll pass the message anyway. + pass + elif self.type == 'multipart': - import multifile - self.multifile = multifile.MultiFile(msg.fp, seekable=False) - self.multifile.push(msg.getparam('boundary')) - self.multifile.next() - self.curfile = self.__class__(self.multifile) + try: + self.multifile = multifile.MultiFile(msg.fp, seekable=False) + self.multifile.push(msg.getparam('boundary')) + self.multifile.next() + self.curfile = self.__class__(self.multifile) + except (TypeError, AttributeError, multifile.Error): + # ignore errors, pass msg as is + self.curfile = msg.fp + self.type = 'binary' if self.type == 'text' or self.type == 'multipart': assert self.curfile is not None + elif self.type == 'binary': + try: + fp.seek(0) + except: + pass + self.curfile = fp else: assert self.curfile is None - + def readline(self): l = '' - if self.type in ('text', 'multipart'): - l = self.curfile.readline() + try: + if self.type in ('text', 'multipart', 'binary'): + l = self.curfile.readline() - if self.type == 'multipart' and not l and self.multifile.next(): - self.curfile = self.__class__(self.multifile) - # recursion. Could get messy if - # we get a bunch of empty multifile parts - l = self.readline() + if self.type == 'multipart' and not l and self.multifile.next(): + self.curfile = self.__class__(self.multifile) + # recursion. Could get messy if + # we get a bunch of empty multifile parts + l = self.readline() + except (TypeError, AttributeError, multifile.Error): + pass return l def next(self): - l = self.readline() - if not l: + try: + l = self.readline() + except multifile.Error, e: + sys.stderr.write("%s: %s\n" % (e.__class__, e)) raise StopIteration - return l + if not l: + raise StopIteration + return l class ClientRunner(object): __slots__ = ['routine', 'all_ok'] - + def __init__(self, routine): self.routine = routine self.setup() - + def setup(self): self.all_ok = True @@ -736,7 +824,7 @@ self.all_ok = False sys.stdout.write(message + str(response.head_tuple()) + '\n') - + class CheckClientRunner(ClientRunner): @@ -750,10 +838,10 @@ self.found_hit = False self.whitelisted = False super(CheckClientRunner, self).setup() - + def handle_response(self, response, message): message += "%s\t" % str(response.head_tuple()) - + if response.is_ok(): wl_count = int(response['WL-Count']) if wl_count > 0: @@ -763,7 +851,7 @@ count = int(response['Count']) if count > 0: self.found_hit = True - + message += "%d\t%d" % (count, wl_count) sys.stdout.write(message + '\n') else: @@ -774,11 +862,11 @@ class InfoClientRunner(ClientRunner): def handle_response(self, response, message): message += "%s\n" % str(response.head_tuple()) - + if response.is_ok(): count = int(response['Count']) message += "\tCount: %d\n" % count - + if count > 0: for f in ('Entered', 'Updated', 'WL-Entered', 'WL-Updated'): if response.has_key(f): @@ -793,9 +881,9 @@ if f is 'WL-Entered': message += ("\tWhiteList Count: %d\n" % int(response['WL-Count'])) - + message += ("\t%s: %s\n" % (f, stringed)) - + sys.stdout.write(message) else: sys.stderr.write(message) @@ -833,13 +921,13 @@ # be removed if self[1] is None: raise ValueError, "no key information" - + for x in self: if not (isinstance(x, long) or x is None): raise ValueError, "Keystuff must be long's or None's" # make sure we didn't get all None's - if not filter(lambda x: x is not None, self): + if not filter(lambda x: x is not None, self): raise ValueError, "keystuff can't be all None's" def from_hexstr(self, s): @@ -869,10 +957,10 @@ class AccountsDict(dict): """Key is pyzor.Address, value is Account When getting, defaults to anonymous_account""" - + anonymous_account = Account((pyzor.anonymous_user, Keystuff((None, 0L)))) - + def __setitem__(self, k, v): typecheck(k, pyzor.Address) typecheck(v, Account) @@ -893,7 +981,7 @@ host : port ; username : keystuff """ __slots__ = ['fp', 'lineno', 'output'] - + def __init__(self, fp): self.fp = fp self.lineno = 0 @@ -906,7 +994,7 @@ while 1: orig_line = self.fp.readline() self.lineno += 1 - + if not orig_line: raise StopIteration line = orig_line.strip() @@ -914,7 +1002,7 @@ continue fields = line.split(':') fields = map(lambda x: x.strip(), fields) - + if len(fields) != 4: self.output.warn("account file: invalid line %d: wrong number of parts" % self.lineno) @@ -939,5 +1027,5 @@ def download(url, outfile): - import urllib - urllib.urlretrieve(url, outfile) + import urllib2 + open(outfile, "wb").write(urllib2.urlopen(url).read())
[-] [+]	Changed	pyzor-0.5.0.tar.bz2/lib/pyzor/server.py ^
@@ -4,7 +4,6 @@ import os import sys -import signal import SocketServer import time import gdbm @@ -17,7 +16,7 @@ __author__ = pyzor.__author__ __version__ = pyzor.__version__ -__revision__ = "$Id: server.py,v 1.25 2002/09/08 03:33:44 ftobin Exp $" +__revision__ = "$Id: server.py,v 1.29 2002-10-09 00:45:45 ftobin Exp $" class AuthorizationError(pyzor.CommError): @@ -343,41 +342,77 @@ -class DBHandle(object): - __slots__ = ['output'] - db_lock = threading.Lock() - max_age = 360024304 # 3 months - db = None +class DBHandle(Singleton): + __slots__ = ['output', 'initialized'] + db_lock = threading.Lock() + max_age = 360024304 # 3 months + db = None + sync_period = 60 + reorganize_period = 3600*24 # 1 day def __init__(self): assert self.db is not None, "database was not initialized" - self.db_lock.acquire() def initialize(self, fn, mode): self.output = Output() self.db = gdbm.open(fn, mode) + self.start_reorganizing() + self.start_syncing() initialize = classmethod(initialize) - def __del__(self): - self.db.sync() - self.db_lock.release() - + def apply_locking_method(self, method, varargs=(), kwargs={}): + # just so we don't carry around a mutable kwargs + if kwargs == {}: + kwargs = {} + self.output.debug("acquiring lock") + self.db_lock.acquire() + self.output.debug("acquired lock") + try: + result = apply(method, varargs, kwargs) + finally: + self.output.debug("releasing lock") + self.db_lock.release() + self.output.debug("released lock") + return result + apply_locking_method = classmethod(apply_locking_method) + def __getitem__(self, key): + return self.apply_locking_method(self._really_getitem, (key,)) + + def _really_getitem(self, key): return self.db[key] def __setitem__(self, key, value): + self.apply_locking_method(self._really_setitem, (key, value)) + + def _really_setitem(self, key, value): self.db[key] = value - def __delitem__(self, key): - del self.db[key] + def start_syncing(self): + self.apply_locking_method(self._really_sync) + self.sync_timer = threading.Timer(self.sync_period, + self.start_syncing) + self.sync_timer.start() + start_syncing = classmethod(start_syncing) - def cleanup(self): - self.output.debug("cleaning up the database") + def _really_sync(self): + self.db.sync() + _really_sync = classmethod(_really_sync) + + def start_reorganizing(self): + self.apply_locking_method(self._really_reorganize) + self.reorganize_timer = threading.Timer(self.reorganize_period, + self.start_reorganizing) + self.reorganize_timer.start() + start_reorganizing = classmethod(start_reorganizing) + + def _really_reorganize(self): + self.output.debug("reorganizing the database") key = self.db.firstkey() breakpoint = time.time() - self.max_age while key is not None: - rec = Record.from_str(self[key]) + rec = Record.from_str(self.db[key]) delkey = None if rec.r_updated < breakpoint: self.output.debug("deleting key %s" % key) @@ -385,14 +420,11 @@ key = self.db.nextkey(key) if delkey: del self.db[delkey] - self.db.reorganize() - + _really_reorganize = classmethod(_really_reorganize) class Server(SocketServer.ThreadingUDPServer, object): - ttl = 4 - timeout = 3 max_packet_size = 8192 time_diff_allowance = 180 @@ -435,8 +467,6 @@ def handle(self): - do_exit = False - try: self._really_handle() except UnsupportedVersionError, e: @@ -447,14 +477,10 @@ # We assume that KeyErrors are due to not # finding a key in the RFC822 message self.handle_error(400, "Bad request: %s" % e) - except TimeoutError, e: - self.handle_error(503, "Gateway timeout: %s" % e) except AuthorizationError, e: self.handle_error(401, "Unauthorized: %s" % e) except SignatureError, e: self.handle_error(401, "Unauthorized, Signature Error: %s" % e) - except SystemExit, e: - do_exit = True except Exception, e: self.handle_error(500, "Internal Server Error: %s" % e) traceback.print_exc() @@ -470,11 +496,6 @@ self.output.debug("sending: %s" % repr(msg_str)) self.wfile.write(msg_str) - if do_exit: - db_hold = DBHandle() # to keep the db consistent - self.finish() - os.kill(self.server.pid, signal.SIGQUIT) - def _really_handle(self): """handle() without the exception handling""" @@ -597,14 +618,9 @@ self.out_msg['WL-Count'] = "%d" % wl_count - def handle_shutdown(self): - raise SystemExit - - dispatches = { 'check': handle_check, 'report': handle_report, 'ping': None, 'info': handle_info, - 'shutdown': handle_shutdown, 'whitelist': handle_whitelist, }
[-] [+]	Changed	pyzor-0.5.0.tar.bz2/scripts/pyzor ^
@@ -1,4 +1,8 @@ -#!/usr/bin/python2 +#!/usr/bin/python + +import os +# set umask +os.umask(0077) import pyzor.client pyzor.client.run()
[-] [+]	Changed	pyzor-0.5.0.tar.bz2/scripts/pyzord ^
@@ -1,9 +1,8 @@ -#!/usr/bin/python2 +#!/usr/bin/python import os import os.path import sys -import signal import getopt import pyzor import pyzor.server @@ -11,20 +10,15 @@ _author__ = pyzor.__author__ __version__ = pyzor.__version__ -__revision__ = "$Id: pyzord,v 1.20 2002/09/08 03:33:44 ftobin Exp $" +__revision__ = "$Id: pyzord,v 1.23 2002-10-09 00:33:44 ftobin Exp $" +progname = 'pyzord' default_anonymous_allows = map(pyzor.Opname, ['check', 'report', 'ping', 'info']) -def cleanup_server_handler(signum, frame): - pyzor.server.DBHandle().cleanup() - -def reopen_log_handler(signum, frame): - server.replace_log(pyzor.server.Log(open(log_fn, 'a'))) - def usage(): - sys.stderr.write("usage: %s [-d] [--homedir dir]\n" % sys.argv[0]) + sys.stderr.write("usage: %s [-d] [--homedir dir]\n" % progname) sys.exit(1) @@ -33,14 +27,14 @@ if os.path.exists(access_fn): pyzor.server.AccessFile(open(access_fn)).feed_into(server.acl) else: - output.warn("%s does not exist; using default ACL; allowing anonymous to do %s" + output.warn("%s does not exist; using default ACL: allowing anonymous to do %s" % (access_fn, default_anonymous_allows)) for op in default_anonymous_allows: server.acl.add_entry(pyzor.server.ACLEntry((pyzor.anonymous_user, op, True))) - + def load_passwd_file(access_fn, server): server.passwd = pyzor.server.Passwd() if os.path.exists(passwd_fn): @@ -51,6 +45,9 @@ ######################################################################## # functions above, run below +# set umask +os.umask(0077) + debug = 0 (options, args) = getopt.getopt(sys.argv[1:], 'dh:', ['homedir=']) if len(args) != 0: @@ -73,8 +70,6 @@ defaults = {'port': '24441', 'listenaddress': '0.0.0.0', - 'logfile': 'pyzord.log', - 'pidfile': 'pyzord.pid', 'digestdb': 'pyzord.db', 'passwdfile': 'pyzord.passwd', 'accessfile': 'pyzord.access', @@ -92,8 +87,6 @@ port = config.getint('server', 'port') listen_adr = config.get('server', 'ListenAddress') -log_fn = config.get_filename('server', 'logfile') -pid_fn = config.get_filename('server', 'pidfile') dbfile = config.get_filename('server', 'DigestDB') passwd_fn = config.get_filename('server', 'passwdfile') access_fn = config.get_filename('server', 'accessfile') @@ -101,48 +94,12 @@ output = pyzor.Output(debug=debug) -logfile = open(log_fn, 'a') pyzor.server.DBHandle.initialize(dbfile, 'c') - server = pyzor.server.Server((listen_adr, port), - pyzor.server.Log(logfile)) + pyzor.server.Log(sys.stdout)) load_passwd_file(passwd_fn, server) load_access_file(access_fn, server) -if not debug: - # This step is required so that the new process is guaranteed - # not to be a process group leader. The next step, setsid(), - # fails if you're a process group leader. - pid = os.fork() - if pid != 0: - os._exit(0) - - # setsid() to become a process group and session group leader. - # Since a controlling terminal is associated with a session, - # and this new session has not yet acquired a controlling - # terminal our process now has no controlling terminal, - # which is a Good Thing for daemons. - os.setsid() - - pid = os.fork() - if pid != 0: - os._exit(0) - - sys.stdin = sys.__stdin__ = open('/dev/null') - sys.stdout = sys.__stdout__ = open('/dev/null', 'a') - #sys.stderr = sys.__stderr__ = open('/dev/null', 'a') - -if pid_fn is not None: - f = open(pid_fn, 'w') - f.write("%d\n" % os.getpid()) - f.close() - -# we shouldn't chdir because that prevents the re-opening -# of logfiles and the database if they are relative paths -##os.chdir("/") - -signal.signal(signal.SIGUSR1, cleanup_server_handler) -signal.signal(signal.SIGHUP, reopen_log_handler) server.serve_forever()
[-] [+]	Changed	pyzor-0.5.0.tar.bz2/scripts/test.sh ^
@@ -3,47 +3,82 @@ # HOME so it finds the right .pyzor export HOME=. export PYTHONPATH=../lib -PYZOR="./pyzor --homedir ." -PYZOR_BOB="./pyzor --homedir bob" -PYZORD="./pyzord --homedir ." -alias check="$PYZOR check < test.in.0" -alias check.bob="$PYZOR check < test.in.0" -kill_server() +start_server() +{ + ./pyzord --homedir . > pyzord.log & + PYZORD_PID=$! + echo "started server with pid $PYZORD_PID" +} + + +pyzor() +{ + ./pyzor --homedir . "$@" +} + + +pyzor_bob() +{ + pyzor --homedir bob "$@" +} + + +check() +{ + pyzor check < test.in.0 +} + + +check_bob() { - echo "killing server" - pidfile='pyzord.pid' - [ -e $pidfile ] && kill `cat $pidfile` + pyzor_bob check < test.in.0 } + +kill_server() +{ + echo "killing server (pid $PYZORD_PID)" + kill $PYZORD_PID +} + fail() { echo "failed: $1" - kill_server exit 1; } setcount() { - count=`$PYZOR check < test.in.0 \| cut -f 3` + count=`pyzor check < test.in.0 \| cut -f 3` } setcount_bob() { - count=`$PYZOR check < test.in.0 \| cut -f 3` + count=`pyzor_bob check < test.in.0 \| cut -f 3` } fail_cmp() { - fail "got $1; expected $2" + fail "got $1; expected $2" } rm -f pyzord.* echo "starting server" -$PYZORD \|\| fail +start_server + +# sleep to give it time to start listening +sleep 2 +[ ! -z $PYZORD_PID ] \|\| fail "we didn't get a process id for the server!" +kill -0 $PYZORD_PID \|\| fail "process is dead" + +trap kill_server 0 + +echo "anonymous: pinging" +pyzor ping \|\| fail echo "anonymous: ensuring a count of 0 at start" setcount @@ -54,19 +89,19 @@ echo "bob: ensuring a count of 0 at start" setcount_bob [ ${count:--1} = 0 ] \|\| fail -check.bob && fail +check_bob && fail echo "anonymous: reporting" -$PYZOR report < test.in.0 && fail +pyzor report < test.in.0 && fail echo "anonymous: reporting" -$PYZOR report < test.in.0 && fail +pyzor report < test.in.0 && fail echo "bob: reporting" -$PYZOR_BOB report < test.in.0 \|\| fail +pyzor_bob report < test.in.0 \|\| fail echo "bob: reporting" -$PYZOR_BOB report < test.in.0 \|\| fail +pyzor_bob report < test.in.0 \|\| fail echo "anonymous: counting reports" @@ -78,38 +113,38 @@ echo "bob: counting reports" setcount_bob [ ${count:--1} = 2 ] \|\| fail_cmp ${count:--1} 2 -check.bob \|\| fail "checking failed" +check_bob \|\| fail "checking failed" echo "bob: reporting a mailbox" -$PYZOR_BOB report --mbox < test.in.mbox \|\| fail +pyzor_bob report --mbox < test.in.mbox \|\| fail echo "bob: counting reports" setcount_bob [ ${count:--1} = 3 ] \|\| fail_cmp ${count:--1} 3 -check.bob \|\| fail "checking exit code failed" +check_bob \|\| fail "checking exit code failed" echo "bob: getting info" # check exit -$PYZOR_BOB info < test.in.0 \|\| fail +pyzor_bob info < test.in.0 \|\| fail # check lines -[ `$PYZOR_BOB info < test.in.0 \| wc -l` = 7 ] \|\| fail +[ `pyzor_bob info < test.in.0 \| wc -l` = 7 ] \|\| fail echo "anonymous: whitelisting" -$PYZOR whitelist < test.in.0 && fail +pyzor whitelist < test.in.0 && fail echo "bob: whitelisting" -$PYZOR_BOB whitelist < test.in.0 \|\| fail +pyzor_bob whitelist < test.in.0 \|\| fail echo "bob: getting info" # check exit -$PYZOR_BOB info < test.in.0 \|\| fail +pyzor_bob info < test.in.0 \|\| fail # check lines -[ `$PYZOR_BOB info < test.in.0 \| wc -l` = 7 ] \|\| fail +[ `pyzor_bob info < test.in.0 \| wc -l` = 7 ] \|\| fail echo "bob: counting reports" @@ -118,16 +153,7 @@ check && fail -echo "anonymous: pinging" -$PYZOR ping \|\| fail - echo "checking for logfile" [ -s pyzord.log ] \|\| fail -echo "anonymous: shutting down server" -$PYZOR shutdown 127.0.0.1:9999 && fail - -echo "bob: shutting down server" -$PYZOR_BOB shutdown 127.0.0.1:9999 \|\| fail - echo "passed"
[-] [+]	Changed	pyzor-0.5.0.tar.bz2/setup.py ^
@@ -12,12 +12,12 @@ distutils.core.setup( name = 'pyzor', version = pyzor.__version__, description = 'networked spam-signature detection', - long_description = long_description, + long_description = long_description, author = 'Frank J. Tobin', author_email = 'ftobin@neverending.org', - license = 'GPL', - platforms = 'POSIX', - keywords = 'spam', + license = 'GPL', + platforms = 'POSIX', + keywords = 'spam', url = 'http://pyzor.sourceforge.net/', scripts=['scripts/pyzor', 'scripts/pyzord'], package_dir = {'': 'lib'},
[-] [+]	Changed	pyzor-0.5.0.tar.bz2/unittests.py ^
@@ -7,7 +7,7 @@ from pyzor.server import * from pyzor.client import * -__revision__ = "$Id: unittests.py,v 1.8 2002/09/07 22:45:03 ftobin Exp $" +__revision__ = "$Id: unittests.py,v 1.8 2002-09-07 22:45:03 ftobin Exp $" class ACLTest(unittest.TestCase):

Changes of Revision 5