Post on 05-Apr-2018
7/31/2019 Auti-Unpack Tricks in Malicious Code
1/30
Anti-unpack Tricks in Malicious Code
Xiaodong TanXiaodong Tan
Security Labs, Websense Inc.Security Labs, Websense Inc.
AVAR 2007, SeoulAVAR 2007, Seoul
7/31/2019 Auti-Unpack Tricks in Malicious Code
2/30
Agenda
Problems raised by packers
Commonly used unpacking methods
Anti static-unpacking tricks
Anti dynamic-unpacking tricks
Conclusions
7/31/2019 Auti-Unpack Tricks in Malicious Code
3/30
Packer Statistics from WSLabs
According to the research of WSLabs (WebsenseSecurity Labs), more than 80% malicious codes aredisguised by a certain packer
Top ten packers from the tracker of WSLabs
7/31/2019 Auti-Unpack Tricks in Malicious Code
4/30
Problems raised by packers
The problems raised by packers
Its easier to produce new malware by usingpackers
Using packers can bring more difficulties to analysts
and researchers ,
Why we need unpack?
In order to detect a known malware, we have todecompress the packer first, and reach the point
where signature matching can occur It is necessary to decompress the file so that the
code analysis or heuristic rules could be applied
7/31/2019 Auti-Unpack Tricks in Malicious Code
5/30
Commonly used unpacking methods
Commonly used unpacking methods
Routine-based (static unpacking)
Emulator-based (dynamic unpacking)
Mixed routine-based and emulator-based
7/31/2019 Auti-Unpack Tricks in Malicious Code
6/30
Routine-based Unpacking
Routine-based unpacking is based on decompressionalgorithms
The packer author always uses standardcompression and decompression algorithms
Algorithm example:
UPack
Advantages: high speed
Disadvantages: Lack of flexibility
7/31/2019 Auti-Unpack Tricks in Malicious Code
7/30
Routine-based Unpacking: A simple model
Packed
Packer
detectionStandard
Algorithms
UnpackedFile
Fileecompress
Re-build PE
7/31/2019 Auti-Unpack Tricks in Malicious Code
8/30
Emulator-based Unpacking
Emulator Unpacking
A file will be loaded and run in the virtual environment
If it can stop right at the OEP, then we can get theunpacked file
Advantages: . ,
different packers
It also can handle the packer with slight modification fromits original version
Disadvantages:
Remarkably affect the performance
In the process of emulation its not an easy thing todetermine when to stop
7/31/2019 Auti-Unpack Tricks in Malicious Code
9/30
Emulator-based Unpacking: A simple model
Packe
dUnpacked
FileVirtualMemory
Virtual CPU
Virtual Loader
VirtualSystem Calls
StopCondition
7/31/2019 Auti-Unpack Tricks in Malicious Code
10/30
Mixed routine-based and emulator-based
Mixed routine-based and emulator-based
If the two types can work harmoniously, we can attainboth efficiency and flexibility
Routine-based unpacking can cope with the standard
decompression algorithms
unknown packers, or find some important parametersfor the routine-based module
7/31/2019 Auti-Unpack Tricks in Malicious Code
11/30
Anti-unpack tricks
Tricks on anti routine-based unpacking
Surviving from packer detection
Modifying the packer inner structure
Modifying the decompression algorithm
Tricks on anti emulator-based un ackin
Interrupting the emulator
Disturbing the emulation stop condition
7/31/2019 Auti-Unpack Tricks in Malicious Code
12/30
Packer detection
Tampering the packer detection is wildly used forescaping the routine-based unpacking
To find out which packer a file has been packed withis usually the first important step in routine-basedunpacking
Usually the packer scanner starts from the entrypoint of a PE file
Its very easy to obfuscate the code around theentry point manually or by tools
A large amount of samples are of this kind and in the
wild
Though this trick is rather simple, if we cant handle thiskind of threat smartly enough, we will leak most of them
7/31/2019 Auti-Unpack Tricks in Malicious Code
13/30
Packer detection
An example of disturbing packer detection
This sample is disguised by a Chinese hacker tool
It can be easily found on almost every Chinesehacker site
The interface of the tool
7/31/2019 Auti-Unpack Tricks in Malicious Code
14/30
Packer detection
Obfuscated stub (Only 16 bytes)
The real ASPack entry point
7/31/2019 Auti-Unpack Tricks in Malicious Code
15/30
Packer structure
Packer structure
Stores a packers important information. Forexample, the information can be as follows:
How to get the decryption key
Where to locate the compressed data
How to reform sections
How to get the OEP
How to fix the import table
And so on
7/31/2019 Auti-Unpack Tricks in Malicious Code
16/30
Packer structure
Take the FSG 2.0 as an example
The entry point of FSG 2.0
7/31/2019 Auti-Unpack Tricks in Malicious Code
17/30
Packer structure
The inner structure of FSG 2.0
7/31/2019 Auti-Unpack Tricks in Malicious Code
18/30
Packer structure
A simple function model for the FSG 2.0 unpackingprocess
void DecompressFSG_2_0(BYTE * image, (void *) LZ77_Decompress)
{
InnerStructAddress = getDWORD(entrypoint + 0x2);
//get the address of the packer structure
DestAddress = getDWORD (InnerStructAddress 0x1C );//get the address where to put the decompressed data
ource a a ress = ge nner ruc ress x
//get the address where to get the compressed data
OriginalEntryPoint = getDWORD (InnerStructAddress + 0x10 );
//get the original entry point
LZ77_Decompress(CompressedDataAddress, TempBuffer);
//decompress the data
CopyBuffer(DestAddress, TempBuffer);
//move the compressed data to the destination
SetEntryPoint(OriginalEntryPoint);
//set the original entry point
}
7/31/2019 Auti-Unpack Tricks in Malicious Code
19/30
Packer structure
Packer structure is usually distant from the entry point (Inthis example, the distance is: 0x101f170 0x1000154 =0x1F01C)
Usually we choose the packer detection signature justaround the entry point
If a packer is detected by a certain packer signature, butits inner structure has been modified, it then cannot getthe right parameters
7/31/2019 Auti-Unpack Tricks in Malicious Code
20/30
Packer structure
A simulation of the packer structure modification
If the offsets in the packer structure are modified, and we still follow the originalprocess, the situation would be as follows:
void DecompressFSG_2_0(BYTE * image, (void *) LZ77_Decompress)
{
InnerStructAddress = getDWORD(entrypoint + 0x2);
//get the address of the packer structure
DestAddress = getDWORD (InnerStructAddress 0x1C);//Wrong parameter!
SourceDataAddress = getDWORD (InnerStructAddress 0x18)
//Wrong parameter!
OriginalEntryPoint = getDWORD (InnerStructAddress + 0x10);
//Wrong parameter!
LZ77_Decompress(CompressedDataAddress, TempBuffer);
//Fail or Crash!
CopyBuffer(DestAddress, TempBuffer);//move the compressed data to the destination
SetEntryPoint(OriginalEntryPoint);
//set the original entry point
}
7/31/2019 Auti-Unpack Tricks in Malicious Code
21/30
Packer algorithm
A routine-based module usually contains many standardalgorithms
Compression/decompression algorithm sometimes canbe modified slightly
If a wrong decompression algorithm is called, there willbe serious problems
To recognize algorithms precisely is very important
7/31/2019 Auti-Unpack Tricks in Malicious Code
22/30
Interrupt emulation
Tricks to interrupt the emulation
FPU (Coprocessor) instructions
MMX (multimedia extension) instructions
Undocumented CPU instructions
SEH Structured Exce tion Handler
One of the most difficult problems during theprocess of emulation
The emulated environment in a product cannothandle all kinds of exception perfectly
7/31/2019 Auti-Unpack Tricks in Malicious Code
23/30
Interrupt emulation: Fake API calling
Another trick about anti-emulation is using fake API calls
Bellow is an example of using fake API call to interruptthe emulator (a variant of the infamous packer Tibs)
7/31/2019 Auti-Unpack Tricks in Malicious Code
24/30
Interrupt emulation: complex program logic
Complex program logic can greatly slow down theemulation process
The performance is usually the bottleneck of anemulator
The worst situation: the emulator quits withoutfinishing the unpacking process (usually a maximumtime-out has been predefined)
Example: using long loops to get a decryption key
There is nothing special in a real environment
The performance may be very low in a virtualenvironment (emulator)
7/31/2019 Auti-Unpack Tricks in Malicious Code
25/30
Emulation stop condition
Possible emulation stop conditions during unpacking
Stop at some API functions
Not called by the packer stub
Have some entry point signatures from standardcompilers
Have some special methods to identify the entrypoint of a certain compiler
Delphi
7/31/2019 Auti-Unpack Tricks in Malicious Code
26/30
Emulation stop condition
The nearest 55 8B EC above the string Runtime erroris the entry point of the Delphi compiled file
7/31/2019 Auti-Unpack Tricks in Malicious Code
27/30
Emulation stop condition
The emulation stop condition can be forged by packer ormalicious code
The stop condition is very important in the emulator-based solution
If the emulation doesnt stop at a correct time, it maymiss the opportunity to get the right unpacked file
7/31/2019 Auti-Unpack Tricks in Malicious Code
28/30
Conclusions
Fighting for the anti routine-based unpacking tricks, weneed:
Detect the packer more accurately and efficiently
Locate the inner structure, and recognize the
decompression algorithms more precisely
handle all kinds of exceptions
Fighting for the anti emulator-based unpacking tricks, weneed:
Improve the emulator performance so that theemulator-based unpacking can be put into practice
Build stronger emulator, in order to get rid of all kindsof anti-emulator tricks
7/31/2019 Auti-Unpack Tricks in Malicious Code
29/30
Conclusions: Handling particularly tricky packers
For some particularly tricky or modified packers
It might be a good strategy to mix emulation andspecific routines
Using emulation to get polymorphic encryption
key, or filter all the garbage instructions
the compression/decompression algorithm isstandard
This solution will bring additional complexity tothe engine
7/31/2019 Auti-Unpack Tricks in Malicious Code
30/30
Q & A
Thank you!