在AMTSO五月份的会议上，来自世界各地的成员一起讨论关于反病毒测试标准的改进，其中，也给出了参加测评的厂商一些指导性意见，通过测评指南，帮助厂商改进产品，进行正确的操作。

主要有如下几点意见：

1. 将重点放在检出率上

2. 标准化的清晰的日志

3. 与测试机构进行更有效地沟通，尤其是在切换设置上

4. 默认设置不会导致弹出窗口

5. 自动与自定义扫描

6. 运行脚本的能力

AMTSO documents are best read in conjunction with the “Fundamental Principles of Testing” and other documents on the AMTSO documents page at http://amtso.org/documents.html. 

Introduction

Focusing on detection rather than performance.

Desiderata

• More accessible logging practice

• Better communication with testers, especially about changes in configuration etc.

• Defaults that don’t cause a pop-up.

• Automation versus customer choice.

• Script ability

Controlling the product: 

• XML configuration file. 

• Global list of actions a product should or might take. 

• If must be signed XML, control must sit with the vendor.

Logging

Clearly a major pain point... 

Format and content:

• what features testers would want log access to, and some way of getting it into a form that can be parsed. 

• Format: conversion of encrypted/odd log formats into a standard format, or instructions on how to convert.

Short term/Long term

• Step 1: what’s required

• Step 2: how to get from current product

• Step 3: get standardization

Vendors could use their obscure log formats, but provide a tool for producing standardized logs. . -> Define standardized log (XML).

What Should Be Logged?

• Event ID 

• Something happened

• What happened

• What caused it (file, URL...)

• Threat ID/classification

• What action/s was/were taken [automated action, prompt to user?]

• Time of event

• Time to take action

Product related content for logging

• Initialization time

• Update time

• Version information

XML view

productinfo

company information

product-specific information

version

signature-version (or versions, where multiple engines used)

update(s) information: last applied, last asked [true/false attribute?]

environment (optional: tester should know this anyway)

OS

CPU

RAM

timestamp

start

finish

object

url

domain

ip-address

path&file @md5 (and other hashes)

process

registry

network (object or event? Product-specific?)

?items within archive

classification/ threatID

malicious-file

malicious-url

Phishing

Dynamic

Spam

IM

Etc. [vendor to classify]

action-taken

blocked

quarantined

disinfected

deleted

user-interaction

action-result

success

failed

[Threat category (if not specified above)]

XML draft from DG:

<product>

       <Name>...</Name>

       <Update>...</Update>

       <OS>...</OS>

       <object>

               <name>...</name>

               <type>File/URL/Firewall</type>

               <date>...</date>

               <detectio>...</detection>

               <actions>

                       <action>

                               <date>...</date>

                               <name>...</name>

                               <result>...</result>

                       </action>

                       ...

                       ...

                       ...

                       <action>

                       </action>

               </actions>

       </object>

</product>

Communication is a major issue

[Favourite quote from Cool Hand Luke goes here. ]

Announcements on product information need to reach testers, not just customers. For example, when a number of products went from pop-ups to toasters while an extensive test was being carried out, it caused significant disruption through need to re-engineer.

A list of all messages and what they mean would be more than useful, but not normally necessary in every possible language. 

Scripts/Macros: it’s easier to ask for/provide information on where windows/buttons etc will be over time, than to ask for product changes. 

Type I or Type II Error? (y/n)

• Which button to press? Advise if popup name changes, if order of buttons changes. Default behaviours.

• Execute or not (y/n). 

• False positive or false negative? y/n. 

• [User will click go-away button if there is one.] 

• Which button to press? Advise if popup name changes, if order of buttons changes. Default behaviours.

• Error messages.

• Vendor incentivized to “confuse” user: force user to decide.

Configuration options: 

• Paranoid

• Secure

• Standard

• ah-what-the-hell...

Standard user profiles

• Paranoid

• “confident” (power-user, super-user)

• home user

• corporate user

• sysadmin

• tester

• drunken user

• Whatever...

Backdoor issues

• Suspend process without killing? Options to reduce security no-go. 

• Provided signed by vendors: tester could create scripts which vendor would sign, but might still be misused.

• “Always allow” creates interesting possibilities

• Manual enabling

Cheat Mode

Tools to aid automation. 

Testing mode/testing variant with extras that aid testing. Response to configuration file. XML?  Dependency issues: what might change in a “special” mode? Security issues? How much do you trust vendors? Trust but verify (forensics after the fact). Tools for controlling behaviour.

Hidden buttons/switches

Tester-friendly product variants

Miscellaneous Thoughts and Further Desiderata

Flexible options. Try to honour or at least respond to requests from testers for changes.

Problems with quarantining: size of quarantine file when large sample set used.

Log file size hard-coded/configurable; allow tester-specified location of log file.

QA Tool sharing