Changeset 364

Show
Ignore:
Timestamp:
08/05/08 20:33:20 (9 years ago)
Author:
cseifert
Message:

merge from 2.2 338-362 to trunk. 2.2 and prior are dead now.

Location:
capture-hpc/trunk
Files:
6 modified

Legend:

Unmodified
Added
Removed
  • capture-hpc/trunk/ReleaseNotes-HPC.txt

    r363 r364  
    22------------- 
    33 
    4 Version 2.2 
     4Version 2.5 
    55----------- 
    66Changes 
    77------- 
    88+ added preprocessor plugin architecture. Preprocessor plugins allow to handle the input urls before they are passed onto capture. For instance, this could be used to create a crawler or filtering plugin. 
    9 + upgraded vmware server to 1.0.6, java 6 update 7, NSIS 2.38 
     9+ added processor ids for state changes (this value is only set if the client plug-in supports this). This allows the client plug-in to determine what URL the attack originates even if multiple URLs are visited at once. The internetexplorerbulk plug-in takes advantage of this functionality. 
     10+ modified client plug-in to communicate the algorithm it is able to support (Divide-and-conquer, bulk, sequential) 
     11+ upgraded vmware server to 1.0.6, java 6 update 7, NSIS 2.38, boost 1.35.0, visual studio 2008 (requires new VC++ Redist Libraries!) 
     12+ removed timeout factor and added absolute timeout/delay config values (see documentation for description of each option) 
     13+ modified tailing of input file; if no more URLs after a specific timeout are detected, the client will stop. 
     14+ implemented staggering revert of virtual machines. If server is configured with multiple VMs, they are not all reverted at the same time. 
     15+ changed threading structure to be more stable (leads to less client inactivity errors) 
     16+ changed IE plugin to close all IE windows (fixes pop ups hanging around) 
     17+ optimized handling of zipping of files - leads to speedup if network capture is not enabled 
     18+ fixed bug 718,729 
    1019 
    1120Known Issues 
     
    1423XXX     When Capture-Client crashes, it will lead to a client inactivity errors. (but those failures are now retried once) 
    1524615     Registry monitoring can't handle a key named  
    16 690     Capture is not able to detect file renames 
    17 676     Empty password on the user of the guest vm in the config.xml causes the capture server to crash (Windows only). 
    18 706     Capture seems to ignore the VM server port. 
     25690 Capture is not able to detect file renames 
     26676 Empty password on the user of the guest vm in the config.xml causes the capture server to crash (Windows only). 
     27706 Capture seems to ignore the VM server port. 
    1928709     sending of exclusion list from server doesnt appear to be working (only affects process exclusion list) 
    20 719     Closing a browser during visitation does not cause this event to be reported back to the server 
     29719 Closing a browser during visitation does not cause this event to be reported back to the server 
    2130721     filedownloader writes to const file name preventing dac algorithm to be applied for applications that make use of this feature 
    22  
    23 Version 2.1.1 
    24 ----------- 
    25 Changes 
    26 ------- 
    27 + added timeout factor that allows one to increase the factor of the various timeouts (e.g. vm revert) 
    28 + implemented staggering revert of virtual machines. If server is configured with multiple VMs, they are not all reverted at the same time. 
    29 + upgraded vmware server to 1.0.5, boost 1.35.0, visual studio 2008 (requires new VC++ Redist Libraries!) 
    30 + changed threading structure to be more stable (leads to less client inactivity errors) 
    31 + changed IE plugin to close all IE windows (fixes pop ups hanging around) 
    32 + optimized handling of zipping of files - leads to speedup if network capture is not enabled 
    33 + fixed bug 718,729 
    3431 
    3532Version 2.1 
  • capture-hpc/trunk/capture-server/Readme.txt

    r363 r364  
    771.Prerequisites 
    88--------------- 
    9 * Sun's Java JRE 1.6.0_02 
    10 * VMWare Server 1.0.5 with VMware VIX (available at http://www.vmware.com/download/server/)  
     9* Sun's Java JRE 1.6.0 - update 7 
     10* VMWare Server 1.0.6 with VMware VIX (do not download VIX separately) (available at http://www.vmware.com/download/server/)  
    1111* Microsoft Windows XP, Microsoft Windows Vista or Linux (other OS might also be capable of running the server, but are not supported) 
    1212 
     
    2424 
    2525* Open up the config.xml 
    26 * Configure the global options, such as the time that is allowed to pass to retrieve a URL, the option to automatically retrieve malware or network captures (on benign and malicious URLs), and the directive to push the local exclusion list to the clients. The timeout_factor allows one to increase the various timeouts (e.g. the number of seconds it takes before a VM is reverted after client fails to connect to server after a revert) 
    27 * The value p_m turns on the divide&conquer feature (see http://www.mcs.vuw.ac.nz/~cseifert/publications/cseifert_divide_and_conquer.pdf) in which client applications are started in parallel and if a malicious state change is detected, the set is divided in half and iterativly visited until the malicious URL is identified (Note there are some risks of missing attacks using this feature; in particular attacks that make use of ip tracking functionality. The paper http://www.mcs.vuw.ac.nz/~cseifert/publications/IFIP2008_CSeifert_Paper66.pdf describes the setup that can reduce this risk. 
    28 The global option group size determines how many instances of the client application are opened at the same time. The lower the value, the more instances are opened. A value of 1 will cause only 1 instance to be opened (just like Capture-HPC v 2.01 and prior). A value of 0.004 will cause 80 (max) instances to be opened. Note only certain client applications support this feature: 
    29         - IE: full support 
    30         - Firefox: full support; however, firefox needs to be configured to open a blank page and not restore from previous sessions. In addition, because firefox does not have a callback that notifies the server when a page has successfully been retrieved, the client-default-visit-time needs to be increased to accommodate loading X firefox instances and retrieving the web pages. Some testing might be required to determine the appropriate value. 
     26* Configure the global options, such as the time that is allowed to pass to retrieve a URL, the option to automatically retrieve malware or network captures (on benign and malicious URLs), and the directive to push the local exclusion list to the clients.  
     27  Various timeout/delay options can also be configured via the global option (all in seconds): 
     28  - client_inactivity_timeout: the capture client indicates that it is still alive via responding to a ping by the server. This happens every 10 seconds. If no pong is received by the client for the duration of the client_inactivity_timeout, the client inactivity error is thrown and the VM reverted. An example when this could happen is when a malicious site causes a blue screen. 
     29  - revert_timeout: the vix code that the revert function makes use of, at times hangs, but functions properly if restarted. If the revert has not completed during the revert_timeout duration, the revert timeout error is thrown and the revert of the VM attempted once again. 
     30  - vm_stalled_after_revert_timeout: identical to the revert_timeout, but the start criteria is not communicated by the VIX api, but rather by the capture client sending a visit command. 
     31  - vm_stalled_during_operation_timeout: When client (e.g. Internet Explorer) locks up, the capture client is still able to respond to pings, but doesnt progress visitation of URLs. This vm_stalled_during_operation_timeout sets how often the capture server should at least expect a visitation event (this is highly dependent on speed of the network and how many URLs are being visited). If no visitation event is received during the timeout period, the VM stalled error is thrown and the VM is reverted. 
     32  - same_vm_revert_delay: the vix library and vmware server have a difficult time reverting vms at the same time. the code already prevents the same VM from reverting at the same time. the delay specified by this variable is automatically applied when reverting the same vm. 
     33  - different_vm_revert_delay: the vix library and vmware server have a difficult time reverting vms at the same time. the delay specified by this variable is automatically applied when reverting a different vm. This delay is larger because theoretically it would be possible to delay two VMs at the same time. 
     34* The global option group size determines how many instances of the client application are opened at the same time. A value of 1 will cause only 1 instance to be opened (just like Capture-HPC v 2.01 and prior). Note only certain client plug-ins support this feature: 
     35        - internetexplorer (applies divide-and-conquer algorithm): full support (max group size of 80) 
     36        - internetexplorerbulk (applies bulk algorithm): full support (max group size of 54) 
     37        - Firefox (applies divide-and-conquer algorithm): full support; however, firefox needs to be configured to open a blank page and not restore from previous sessions. In addition, because firefox does not have a callback that notifies the server when a page has successfully been retrieved, the client-default-visit-time needs to be increased to accommodate loading X firefox instances and retrieving the web pages. Some testing might be required to determine the appropriate value. 
    3138        - Other: not supported at this point 
    3239* Add the local exclusion lists that would be pushed to the clients if that option is enabled  
     
    4552                        send-exclusion-lists="false" 
    4653                        group_size="50" 
    47                         timeout_factor="1.0" 
     54                                vm_stalled_after_revert_timeout="120" 
     55                revert_timeout="120" 
     56                client_inactivity_timeout="60" 
     57                vm_stalled_during_operation_timeout="300" 
     58                same_vm_revert_delay="6" 
     59                different_vm_revert_delay="24" 
    4860        /> 
    4961 
  • capture-hpc/trunk/capture-server/capture/VMwareServer.java

    r328 r364  
    4141    private String password; 
    4242    private int uniqueId; 
    43     private static int REVERT_TIMEOUT = 120; 
     43    private static int REVERT_TIMEOUT = (1000 * Integer.parseInt(ConfigManager.getInstance().getConfigOption("revert_timeout"))); 
    4444 
    4545    private LinkedList<VirtualMachine> virtualMachines; 
     
    5555        this.username = username; 
    5656        this.password = password; 
    57         if(System.getProperty("fixIds") != null && System.getProperty("fixIds").equals("true")) { 
     57        if (System.getProperty("fixIds") != null && System.getProperty("fixIds").equals("true")) { 
    5858            uniqueId = 1; 
    5959        } else { 
     
    121121                        class VixThread extends Thread { 
    122122                            public int returnCode = 1; 
    123                             Process vix = null; 
     123                            Process vix = null; 
    124124 
    125125                            public void run() { 
     
    137137                                    returnCode = 17; //VIX_TIMEOUT 
    138138                                    if (vix != null) { 
    139                                         System.out.println("vix null"); 
     139                                        System.out.println("vix null"); 
    140140                                        try { 
    141141                                            String line = stdInput.readLine(); 
    142142                                            while (line != null) { 
    143                                                 System.out.println("line"); 
     143                                                System.out.println("line"); 
    144144                                                System.out.println(line); 
    145145                                                line = stdInput.readLine(); 
    146146                                            } 
    147                                                                                 System.out.println("line null"); 
     147                                            System.out.println("line null"); 
    148148                                        } catch (Exception ef) { 
    149149                                            System.out.println(ef.getMessage()); 
     
    186186                            vixThread.interrupt(); 
    187187                        } 
    188                         vixThread.join(1000); //wait for a chance of for thread to finish 
     188                        vixThread.join(1000); //wait for a chance of for thread to finish 
    189189 
    190190                        int error = vixThread.returnCode; 
    191191 
    192                         synchronized(item.vm) { 
     192                        synchronized (item.vm) { 
    193193                            if (error == 0) { 
    194194                                item.vm.setLastContact(Calendar.getInstance().getTimeInMillis()); 
     
    204204                                //identical VM. Occurs, for example if malicious URLs are encountered; dont slow things down much 
    205205                                System.out.println("Reverting same VM...just waiting a bit"); 
    206                                 Thread.sleep((long) (6000 * Double.parseDouble(ConfigManager.getInstance().getConfigOption("timeout_factor")))); 
     206                                Thread.sleep(1000 * Integer.parseInt(ConfigManager.getInstance().getConfigOption("same_vm_revert_delay"))); 
    207207                            } else { 
    208208                                System.out.println("Reverting different VM...waiting considerably"); 
    209209                                //reverting different VMs (for instance during startup); this needs to be throttled considerably 
    210                                 Thread.sleep((long) (24000 * Double.parseDouble(ConfigManager.getInstance().getConfigOption("timeout_factor")))); 
     210                                Thread.sleep(1000 * Integer.parseInt(ConfigManager.getInstance().getConfigOption("different_vm_revert_delay"))); 
    211211                            } 
    212212                            lastVM = item.vm.getVmUniqueId(); 
  • capture-hpc/trunk/capture-server/capture/VirtualMachinesStateChecker.java

    r353 r364  
    2222                    if (vm.getState() == VM_STATE.RUNNING) { 
    2323                        long diff = currentTime - vm.getLastContact(); 
    24                         if (diff >= (60000 * Double.parseDouble(ConfigManager.getInstance().getConfigOption("timeout_factor")))) { 
     24                        if (diff >= (1000 * Integer.parseInt(ConfigManager.getInstance().getConfigOption("client_inactivity_timeout")))) { 
    2525                            Stats.clientInactivity++; 
    2626                            System.out.println(vm.getLogHeader() + " Client inactivity, reverting VM"); 
     
    3737 
    3838                        diff = currentTime - vm.getTimeOfLastStateChange(); 
    39                         if (diff >= (500000 * Double.parseDouble(ConfigManager.getInstance().getConfigOption("timeout_factor")))) { 
     39                        if (diff >= (1000 * Integer.parseInt(ConfigManager.getInstance().getConfigOption("vm_stalled_during_operation_timeout")))) { 
    4040                            Stats.vmStalled++; 
    4141                            System.out.println(vm.getLogHeader() + " VM stalled during operation, reverting VM"); 
     
    5555                        } 
    5656                        long diff = currentTime - vm.getTimeOfLastStateChange(); 
    57                         if (diff >= (300000 * Double.parseDouble(ConfigManager.getInstance().getConfigOption("timeout_factor")))) { 
     57                        if (diff >= (1000 * Integer.parseInt(ConfigManager.getInstance().getConfigOption("vm_stalled_after_revert_timeout")))) { 
    5858                            Stats.vmStalled++; 
    5959                            System.out.println(vm.getLogHeader() + " VM stalled, reverting VM"); 
  • capture-hpc/trunk/capture-server/config.xml

    r340 r364  
    88                        send-exclusion-lists="false" 
    99                group_size="50" 
    10                         timeout_factor="1.0" 
    11         /> 
     10                        vm_stalled_after_revert_timeout="120" 
     11            revert_timeout="120" 
     12            client_inactivity_timeout="60" 
     13            vm_stalled_during_operation_timeout="300" 
     14            same_vm_revert_delay="6" 
     15            different_vm_revert_delay="24" 
     16    /> 
    1217         
    1318        <exclusion-list monitor="file" file="FileMonitor.exl" /> 
  • capture-hpc/trunk/capture-server/config.xsd

    r340 r364  
    5858                <xs:attribute name="send-exclusion-lists" type="xs:boolean" use="required"></xs:attribute> 
    5959            <xs:attribute name="group_size" type="xs:integer" use="required"></xs:attribute> 
    60                         <xs:attribute name="timeout_factor" type="xs:double" use="required"></xs:attribute> 
    61         </xs:complexType> 
     60                        <xs:attribute name="client_inactivity_timeout" type="xs:integer" use="required"></xs:attribute> 
     61            <xs:attribute name="revert_timeout" type="xs:integer" use="required"></xs:attribute> 
     62            <xs:attribute name="vm_stalled_after_revert_timeout" type="xs:integer" use="required"></xs:attribute> 
     63            <xs:attribute name="vm_stalled_during_operation_timeout" type="xs:integer" use="required"></xs:attribute> 
     64            <xs:attribute name="same_vm_revert_delay" type="xs:integer" use="required"></xs:attribute> 
     65            <xs:attribute name="different_vm_revert_delay" type="xs:integer" use="required"></xs:attribute> 
     66        </xs:complexType> 
    6267    </xs:element> 
    6368</xs:schema>