You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449
  1. SNF MDaemon Plugin Change Log...
  2. ------------------------------------------------------------------------------
  3. 20080626 - Version 3.0, It's official.
  4. Changed build information.
  5. Removed extraneous comments from configuration file.
  6. 20080524 - Version V2-9rc6.25.7
  7. Optimized networking library for additional speed & stability by moving
  8. receive buffer allocation from heap to stack (automatic).
  9. Optimized timing parameters in SNFClient for improved speed. Polling dealys
  10. are now reduced to 10ms from 30ms.
  11. Removed speed-bug in SNFClient, 100ms guard time between retries was always
  12. executed after an attempt (even a successful attempt). The guard time is now
  13. condition and only fires on unsuccessful attempts.
  14. Updated XCI server logic to ensure non-blocking sockets for clients in all
  15. socket implementations.
  16. 20080424 - Version V2-9rc6.24.6
  17. Refactored snfScanData.clear() to reduce heap work and fragments.
  18. Added mutex to scanMessageFile() entry point just in case some app attempts to
  19. put multiple threads through a single engine handler. scanMessage() is already
  20. protected and fully wraped by the new scanMessageFile() mutex.
  21. Added non-specific runtime exception handling to XHDR injection code.
  22. Added 2 retries w/ 300ms delay to remove original message in XHDR inject code.
  23. If remove fails after 3 attempts the injector throws.
  24. Added 2 retries w/ 300ms delay to rename temp file to msg in XHDR inject code.
  25. If rename fails after 3 attempts the injector throws.
  26. Added IPTest logging.
  27. 20080416 - Version V2-9rc5.23.6
  28. Fixed bug where SNCY open() would fail on some Win* platforms with
  29. WSAEINVAL instead of the standard EINPROGRESS or EALREADY which were expected.
  30. Also added WSAEWOULDBLOCK to cover other "ambiguities" in windows sockets
  31. implementations. InProgress() on Win* now test for any of:
  32. WSAEINPROGRESS, WSAEALREADY, WSAEWOULDBLOCK, WSAEINVAL
  33. 20080413 - Version V2-9rc5.22.6
  34. Fixed bug in TCPHost.open() where EALREADY was not counted as a version of
  35. EINPROGRESS. This would cause open() to throw an unnecessary exception when
  36. an open() required extra time.
  37. 20080413 - Version V2-9rc5.21.6
  38. Extended timeout for SYNC session open() to the full session length. This way
  39. if a session takes a long time to open it still has a shot at success.
  40. 20080411 - Version V2-9rc5.20.6
  41. Adjusted snfNETmgr to use non-blocking open in SYNC sessions. Open timeout
  42. is 1/3 of the session timeout. Session timeout is 2 * Session pacing. Open
  43. polling uses golden spiral delay from 10ms to 340ms.
  44. 20080410 - Version V2-9rc5.19.6
  45. Adjusted XCI manager to use new snfCFGPacket paradigm in checkCFG().
  46. Adjusted snf_RulebaseHandler::addRulePanic() to use MyMutex and eliminated
  47. the AutoPanicMutex and waiting scheme.
  48. Refactored scanMessage() to use a ScopeMutex() rather than lock()/unlock().
  49. Refactored scanMessage() to use MyCFGPacket.isRulePanic() test.
  50. Redesigned snfCFGPacket handling to automate grab() / drop() functions.
  51. Fixed lock-up bug: Redesigned AutoPanic posting and checking mechanisms to
  52. eliminate potential dead-lock condition. Under some conditions a precisely
  53. timed auto-panic posting could cause the RulebaesHandler mutex and the
  54. AutoPanicMutex to become intertwined leading to a cascading deadlock. When
  55. this occurred all XCI processing threads and eventually the XCI listener
  56. thread would become blocked waiting to get the current configuration.
  57. 20080409 - Version V2-9rc5.18.6
  58. Enhanced XCI exception handling and logging to provide additional detail.
  59. Added code to explicitely check for zero length files in scanMessagFile().
  60. Previously a zero length file would cause the CBFR module of the filter
  61. chain to throw an invalid buffer exception. Now if the message file is empty
  62. scanMessageFile() will throw a FileError stating FileEmpty!.
  63. 20080407 - Version V2-9rc5.17.6
  64. Enhanced exception reporting in snfXCImrg
  65. 20080405 - Version V2-9rc5.16.6
  66. Reduced safetly limits on status reports to 100K for status reports and 100K
  67. for samples. Previous values were 10M. Most full sessions from the busiest
  68. systems are < 50K total.
  69. Recoded sendDataTimeout() to break uploads into 512 byte chunks and insert
  70. delays only when a chunk is fragmented. This methodology improves reliability
  71. on Win* systems without any significant penalty on systems that don't need
  72. socket sends() to be in smaller chunks.
  73. Fixed TCPClient::transmit() and TCPHost::transmit() bug where returned byte
  74. count might be -1. Now returned byte counts can only be 0 or more.
  75. 20080403 - Version SNF2-9vr5.15.5
  76. Minor modifications to networking module to better support non-blocking open()
  77. Updated SNFClient with new timing and non-blocking open(). Worst case return
  78. time from SNFClient estimated at 200 seconds (theoretically impossible). No-
  79. connection return time from SNFClient estimated at 20 seconds.
  80. 20080326 - Version SNF2-9rc4.15.4
  81. Refactored snfNETmgr::sync() to consolidate non-blocking io routines.
  82. Added detailed thread status data to XCI listener thread.
  83. Refactored snfNETmgr::sync() to check a Timeout, removed TCPWatchdog.
  84. 20080325 - Version SNF2-9rc4.12.4
  85. Added a "Rulebase Getter" feature as part of the snf_Reloader. When enabled
  86. the Rulebase Getter will launch a user defineable system() call whenever a
  87. new rulebase file is available. The call will be repeated until the condition
  88. is cleared by a successful update of the rulebase file. The Rulebase Getter
  89. will wait a configurable "guard time" between attempts. The default system()
  90. call is "getRulebase" with a guard time of 3 minutes. In most cases this will
  91. launch the provided getRulebase script which should be present in the start
  92. location of SNFServer on most systems. Best practice is to configure the full
  93. path to the update script. The system() call is made in a separate thread so
  94. that if the system() call hangs for some reason only the Rulebase Getter is
  95. stuck.
  96. Improved exception handling/reporting in scanMessageFile().
  97. Updated scanMessagFile() header injection code to accommodate messages with
  98. no body. Previous version would throw an exception when it could not find an
  99. injection point. The new version makes the injection point byte 0 and puts
  100. the injected headers at the top of the message using it's best guess about the
  101. type of line endings (CRLF or LF) to use.
  102. Updated Networking library to use SO_REUSEADDR by default on listeners.
  103. 20080319 - Version SNF2-9rc4.11
  104. Added IPScan on-off to snfmdplugin.xml. This allows users to turn off the
  105. IPScan feature without editing the Plugins.dat file as was previously
  106. required. The feature can now be enabled or disabled at will by editing the
  107. configuration file.
  108. Added Configuration editor options to snfmdplugin.xml. Previously the built-
  109. in configuration function was hard coded to start notepad with the config
  110. file. Now the system() call made by the ConfigFunc() can be edited in the
  111. configuration file. The configuration file name can be appended to the
  112. command optionally. The default is still to start notepad and append the
  113. configuration file path so that it is loaded automatically. It is hoped that
  114. GUI based configuration editors for the SNF plugin will be built by third
  115. parties and in the mean time folks can now configure their favorite XML file
  116. editor to modify their SNF plugin configuration.
  117. Modified API use fixed shutdown bug - The plugin used to initialize the SNF
  118. scanning engine when the DLL was loaded and would shut it down when the DLL
  119. was unloaded. Now the Startup and Shutdown functions in the MDaemon plugin
  120. API. This ensures that the engine components are started and shutdown in the
  121. proper sequence.
  122. Included new SNFEngine core (excerpts from that change log included).
  123. 20080318 - SNF2-9rc1.11.exe Consolidated several mods/fixes
  124. Corrected scan error logging bug. Was posting <s/> now posts <e/>.
  125. Updated scan error logging to be more uniform with non-scan errors.
  126. Developed various script prototypes for postfix integration & automated
  127. updates on win* systems using the new UpdateReady.txt file mechanism.
  128. Fixed a bug in scanMessageFile() where an \n\n style insertion point
  129. would never be detected.
  130. Modified scanMessageFile() header injection to strip <CR> from line ends
  131. when the message file provided does not use them. The line-end style of
  132. the message file is detected while locating the insertion point. If the
  133. insertion point (first blank line) does not use <CR><LF> then the SNF
  134. generated X-Headers are stripped of <CR> in a tight loop before injection.
  135. Enhanced error and exception reporting in SNFMulti.cpp scanMessageFile().
  136. Enhanced exception handling in networking module. All exceptions now
  137. throw descriptive runtime_error exceptions.
  138. 20080306 - SNF2-9rc1.8.exe (FIRST RELEASE CANDIDATE for VERSION 3!)
  139. Added Drilldown Header Directive Functions - When the candidate source IP
  140. comes from a header matching a drilldown directive the IP is marked "Ignore"
  141. in GBUdb and the candidate is no longer eligible to be the source for that
  142. message. This allows SNF to follow the trusted chain of devices (by IP) down
  143. to the actual source of the message. It is handy for ignoring net blocks
  144. because it can match partial IPs but it is designed to allow SNF to learn
  145. it's way through the servers at large ISPs so that the original source for
  146. each message can be evaluated directly.
  147. Added Source Header Directive Functions - This feature allows SNF to acquire
  148. the source IP for a message from a specific header rather than searching
  149. through the Received headers in the message. This is useful when the original
  150. source for a message is not represented in Received headers. For example:
  151. Hotmail places the originating source IP in a special header and does not
  152. provide a Received header for that IP. This feature is protected from abuse
  153. by a "Context" feature which only activates the source header directive when
  154. specific content is found in a specific received header. Using the above
  155. example, this feature can be configured so that a Hotmail source header would
  156. only be read if the top Recieved header contained "hotmail.com [" indicating
  157. that the ptr lookup for the header matched the hotmail domain. Note: When a
  158. source is pulled from a header directive that source is put into a synthetic
  159. Received header and injected into the scanning stream (not the message) as
  160. the first Received header.
  161. Added forced source IP to XCI - It is now possible to "inject" or "force"
  162. the source IP for any message by providing that IP in the XCI request or
  163. directly in a scan...() function call. This allows the calling application
  164. to provide the source IP for a message ahead of any Received headers that
  165. might be in the message. This is useful when the calling application knows
  166. the original source IP for the message but that IP is not represented in
  167. the Received headers and it is not desireable to use the Source Header
  168. Directive mechanism.
  169. Added forced source IP mode to SNFClient - It is now possible to call the
  170. SNFClient utility with an IP4Address using the syntax:
  171. SNFClient -source=12.34.56.78
  172. The -source mode of SNFClient exercises the forced source IP feature in
  173. the XCI (see above)
  174. Added Status Report features to SNFClient and XCI - It is now possible to
  175. request the latest status.second, status.minute, or status.hour data via
  176. the XCI and SNFClient. The syntax for requesting a status report using the
  177. SNFClient is:
  178. SNFClient -status.second
  179. SNFClient -status.minute
  180. SNFClient -status.hour
  181. In addition to providing status reports the SNFClient in this mode will
  182. return a nonzero value (usually 99) if it is unable to get a status report
  183. from SNFServer. This feature can be used to verify that SNFServer is up
  184. and responding. If SNFServer is OK then the result code returned is 0.
  185. Added result codes to SNFClient -test and XCI IP test functions - The XCI
  186. engine has been upgraded to provide the range value for the IP under test
  187. as well as the symbolic result code associated with that range. This allows
  188. the -test function to provide results that are consistent with the GBUdb
  189. configuration without additional processing: For example, if the IP falls
  190. in the Caution range then the Caution result code will be returned just
  191. as if a message had been scanned with the same IP and no pattern match
  192. occurred. The same is true for Truncate and Black range hits.
  193. Added Timestamp and Command Line Parameter data to SNFClient.exe.err - When
  194. an error occurs with SNFClient that may not appear in the SNFServer logs an
  195. entry is appended to the SNFClient.exe.err file. That in itself is not new.
  196. The new feature is that the entries added to the SNFClient.exe.err file now
  197. include timestamp and command line data to aid in debugging.
  198. Updated the Configuration Log to include all of the current configuration
  199. features and to improve it's readability.
  200. 20080207 - SNF2-9b1.7.exe
  201. SYNC Timeout now 2x SYNC Schedule
  202. SNFServer now produces an UpdateReady.txt file when the UTC timestamp on
  203. the SYNC server is newer than the UTC timestamp of the active rulebase. It
  204. is presumed that a suitable update script or program will run periodically
  205. and download a fresh rulebase file if the UpdateReady.txt file is present.
  206. The update script should remove the UpdateReady.txt file when it completes
  207. a successful download of the new rulebase file.
  208. Added available rulebase UTC in status reports <udate utc.../>
  209. Added Automatic path fixup for ending / or \
  210. Added option to use local time in log rotation <rotation localtime='no'/>
  211. The default is still utc.
  212. 20071102 - SNF2-9b1.6.exe
  213. Increased MAX_EVALS from 1024 to 2048.
  214. Adjusted defult range envelopes in snf_engine.xml to be more conservative.
  215. 20071017 - Version SNF2-9b1.5
  216. Added a missing #include directive to the networking.hpp file. The
  217. missing #include was not a factor on Linux and Windows systems but
  218. caused compiler errors on BSD systems.
  219. Corrected a bug in the GBUdb White Range code where any message with a
  220. white range source IP was being forced to the white result code. The
  221. engine now (correctly) only forces the result and records the event when
  222. a black pattern rule was matched and the White Range IP causes that
  223. scan result to be overturned. If the scan result was not a black pattern
  224. match then the original scan result is allowed to pass through.
  225. Corrected a bug in the Header Analysis filter chain module that would
  226. cause the first header in the message to be ignored in some cases.
  227. Corrected an XML log format problem so that <s/> elements are correctly
  228. open ended <s ....> or closed (empty) <s..../> according to whether they
  229. have subordinate elements.
  230. Adjusted the GBUdb header info format. The order of the Confidence
  231. figure and Probabilty figure is now the same as in the XML log files
  232. (C then P). The confidence and probability figures are now preceeded
  233. with c= and p= respectively so that it's easy to tell which is which.
  234. 20071009 Version 2-9b1.4
  235. Tightened up the XCI handler code and removed the watchdog. The watchdog
  236. would restart the listener if there were no connections in 5 minutes. It
  237. was originally added to provide additional stability, however in practice
  238. there have been no "stalled listeners". Also, a stalled listener would
  239. likely be a sign of a different problem that the watchdog would tend to
  240. hide.
  241. Modified and refactored the XCI configuration management code. All XCI config
  242. changes and up-down operations are now handled in a single function except
  243. upon exit from the main XCI thread where XCI_shutdown() is always called.
  244. Added some more detailed exception handling code to the XCI component so that
  245. more data will be logged in the event of an error.
  246. Reviewed and modified the InstallInstructions.txt file. Removed this log
  247. to this separate file.
  248. Modified the snfmdplugin.xml file to properly configure the new features in
  249. the engine.
  250. * Header training directives and new <training/> section.
  251. * XCI interface configuration.
  252. * Tweaks to GBUdb ranges.
  253. * msg-file type configuration (not used in MDaemon, but configured anyway)
  254. ----
  255. Version 2-9a11 (engine a53)
  256. * Enhanced IP extraction from Received headers so that any unexpected bytes
  257. between the [ and ] will force the attempt to be aborted.
  258. * Fix the IP test code so that the IP 0.0.0.0 cannot be the source IP and
  259. cannot be tested.
  260. Version 2-9a11 (engine a52)
  261. * Corrected plug-in log entry logic. Allowed/Rejected tag now comes directly
  262. from the message rejection logic and is accurate in all cases.
  263. Version 2-9a10 (engine a52)
  264. * Corrected a bug in the MessageIPFunc where Ignore flagged IPs would still
  265. cause rejected messages if the statistics were in the Truncate range. Now
  266. messages are rejected in only two cases:
  267. The Flag is _Ugly_ and the rating is _Truncate_ or, the Flag is _Bad_.
  268. Version 2-9a9 (engine a52)
  269. * Adjusted IPtest module in HeaderAnalysis to handle TooManyIPs exception
  270. locally and silently.
  271. * Increased HeaderAnalysis IP limit from 20 to 50.
  272. Version 2-9a9 (engine a51)
  273. * Corrected possible heap corruption bug in EvaulationMatrix Destructors.
  274. * Added trace strings to scanMessage() for tighter panic reporting.
  275. * Added caching to snf_engine Evaluator allocation scheme.
  276. * Added optimizations to snf_engine Evaluator safety checks.
  277. Version 2-9a8
  278. * Added deep exception handling to Token Matrix objects.
  279. Version 2-9a7
  280. * Exception handling throughout the engine has been refactored to use std:exception
  281. and to provide additional detail via e.what()
  282. * The plug-in log will now show e.what() data as SNF Debug: whenever an exception
  283. is thrown during a message scan.
  284. Version 2-9a6
  285. * Adjusted .ctl file path converter to accept either .msg or .tmp paths.
  286. Version 2-9a5
  287. A lot of new things were learned, updated, and corrected.
  288. * Fixed the "lockup" when the plugin failed to start successfully. The cause of this
  289. appears to be a threading issue associated with DLLs that are being initialized.
  290. If threads are created during the initialization of a DLL, the DLL must succeed!
  291. The threads that are created do not get any cycles until after the DLL is loaded
  292. successfully. As a result, if the initialization process attempts to join() these
  293. threads a deadlock is created. The fix was to allow the SNF plugin initialization
  294. process to succeed in all cases while setting a flag that forces the engine to
  295. be inert if the initialization was not successful. When the DLL is later unloaded
  296. the threads are already running so the join() calls that are part of the engine
  297. cleanup code are able to complete without incident.
  298. * Installed detailed exception handling for the start-up sequence. The plugin can
  299. now report on very specific reasons for failing to initialize properly.
  300. * Fixed a bug in the GBUdbIgnoreList processor where long lines would cause the
  301. remainder of the file not to be read. The line length limit still exists, but
  302. it is now 255 characters which is unlikely to occur and would be considered
  303. incorrect formatting.
  304. * The threading library now includes top-level exception handling to trap any
  305. exception that was not handled by myTask(). Along with this two flags were
  306. added to thread objects: isRunning() and isBad(). isRunning() is true when a
  307. thread object is still active. isBad() is true if the thread failed to start or
  308. an exception escaped myTask().
  309. * At least one GBUdbIgnoreList entry is now REQUIRED. If the count of IPs from the
  310. GBUdbIgnoreList.txt file is less than 1 (or the file is missing) then the plug-in
  311. will complain and fail to start.
  312. * snf2check.exe has been removed from the distribution for the time being since it
  313. causes some systems to strip the attachment or block the email. This is the same
  314. program that is already on existing SNF systems.